serhii.net

Obsidian for zettelkasten-type notes

.. is probably my new obsession, along with getting it to play nicely with Hugo. It’s a closed non-open-source system but files are saved as markdown, has an awesome Android app - everything I’ve ever wanted except openness, basically.

So:

Template to create hugo-compatible front matter in Obsidian:

Templater¹ is a community plugin for template stuff, but supports neat things like getting clipboard data, creating files, etc. Additionally supports automatically using templates when creating notes in a folder or in general and a lot of other excellent stuff.

This template gets run manually after I create and name a note. When I run it, it autogenerates Hugo front matter, gets the title from the filename, and puts the cursor in the first tag. The second tag is created from the folder name where the note is located, currently I defined two: it and rl.

---
title: "<% tp.file.title %>"
tags:
  - "zc"
  - "zc/<% tp.file.folder() %>"
  - "<% tp.file.cursor() %>"
fulldate: <% tp.date.now("YYYY-MM-DDTHH:MM:SSZZ") %>
date: <% tp.date.now("YYYY-MM-DD") %>
hidden: false
draft: true
---

Obsidian to Hugo Conversion

I looked at zoni/obsidian-export: Rust library and CLI to export an Obsidian vault to regular Markdown and khalednassar/obyde: A minimal tool to convert a standardly configured Obsidian vault to a Jekyll or Hugo blog., found the latter to be a bit clearer in how it handles assets etc. It requires a date in frontmatter in YYYY-MM-DD format, which I provided.

round() has weirdly unexpected behaviour that I’m ashamed I didn’t notice or know about:

if two multiples are equally close, rounding is done toward the even choice (so, for example, both round(0.5) and round(-0.5) are 0, and round(1.5) is 2) ¹

So:

>>> round(1.5)
2
>>> round(2.5)
2
>>> round(3.5)
4

Python math.isclose() to check for “almost equal”

Had an issue with checking whether a sum of floats sums up to a number, remembering that python floats are ‘special’:

>>> 0.1 + 0.2
0.30000000000000004

Stack overflow¹ told me about math.isclose(), works as you’d expect:

assert math.isclose(sum(floats), needed_sum)

unittest skip test based on condition

From unittest documentation ¹

class MyTestCase(unittest.TestCase):

    @unittest.skipIf(mylib.__version__ < (1, 3), "not supported in this library version")
    def test_format(self):
        # Tests that work for only a certain version of the library.
        pass

Google Meet

You can minimize your own video, and then make the entire window much smaller!

Python strings formatting

Obvious, but: you can declare strings and format them in separate places!

constants.py:

my_string = "Hello my name is {0}"

other_file.py:

from constants import my_string
print(my_string.format("Serhii"))

Pycharm run current unittest binding

<C-S-F10> runs the unittest where the cursor is currently located. Or all of them if located anywhere else in the file.

TODO: set binding to do the same, but debugging.

python - run only some tests from a test suite

I wanted to run only some test files, all except the ones where I needed a GPU. Wrote this:

import subprocess

# Parts of filenames to exclude
large_tests = ['component', 'test_temp']

test_folder = Path(__file__).parent.absolute()
test_files = list(test_folder.glob("test_*.py"))
test_files = [x.name for x in test_files]

for l in large_tests:
  test_files = list(filter(lambda x: l not in x, test_files))

commands = ["python3", "-m", "unittest"] + test_files

subprocess.run(commands, cwd=test_folder)

Notes:

• Thought this would be a security nightmare, but it’s not¹ - unless shell=True is explicitly passed, no shell is called, ergo no shell-command-injection stuff is possible.
• os.chdir() is nicely replaced by the cwd= parameter, much nicer than what I’d have done previously!

Python typing annotating second-order functions

def my_function(other_function: Callable) -> Callable:
  return other_function

Pycharm run all unit tests in a folder

What I’d do as

cd tests
python3 -m unittest

in Pycharm is right-clicking on a directory in Project view and “Run unittests”

OOP principles

Open/Closed principle: you should be able to open a module/class to add stuff easily, but otherwise you shouldn’t need to touch it for existing stuff.

Python dir

Wrote a line like if dir is not None .., but dir is a builtin! It returns all the names in the current scope.

Pycharm debugging

You can add Watches, values that will be shown and tracked! Nice for debugging stuff that needs values that are deep in other variables

Python unittests run at the end of the class/module

• class-level:

  • setUpClass(cls) gets called before tests from one class get run, not once per test
  • tearDownClass(cls) gets called before tests from one class get run, not once per test
  • Both need to be class methods, a la:¹

      class Test(unittest.TestCase):
          @classmethod
          def setUpClass(cls):
              cls._connection = createExpensiveConnectionObject()
    

• module-level

  • setUpModule(), tearDownModule()
  • should be implemented as normal functions

  Aaanad if you set any class variables, you can still access them as self.xxx from within the tests!

Python or in arguments

Neat thing seen in detectron default_argument_parser:

def argparser(epilog=None):
  ...
  x = epilog or "here's some text"

Where “here’s some text” is a long string that doesn’t really belong in the function signature.

A really nice pattern, much better than my usual

if x is None:
  x = ...

vim open list of files from cli

vim -p `ag -l whatever`

opens each file returned by ag. (ag -l lists only the files with matches and nothing else)

Replacing jekyll-style highlight tags with standard markdown ones

In some posts I had code blocks like {% highlight html %} etc. The html/js got parsed, and some “here’s how to redirect using javascript” code got executed in the master page.

Here’s how I replaced all that syntax with the standard markdown one:

for f in `ag -l "endhighlight"`;
do cat $f | sed "s/{% highlight \(.*\) %}/\`\`\`\1/" | sed "s/{% endhighlight %}/\`\`\`/g" > $f;
done

Python dataclasses and classmethods

@dataclass
class MyClass:
  x: int = 4

@classmethod
def init_whatever(number: int)
  return cls(x=number)

Python exceptions and unittests

unittest’s self.assertRaisesRegex() is nice but couldn’t get it to work with my custom exception class.

with self.assertRaisesRegex(CustomException, "No trained model"):

It expects the message to be in e.args¹. args also gets used by the Exception class for __str__() etc, so it’s a nice thing.

Set it up easily:

class CustomException(Exception):
    def __init__(self, detailed_message: str = None):
        if detailed_message:
          self.detailed_message = detailed_message
          self.args = (self.detailed_message, )

Catching python regex exceptions

try:
  re.search("DataLoader worker(.*is killed by signal: Bus error", text)
except re.error:
  whatever()

TODO I really like this regex tutorial: Regular Expressions: Regexes in Python (Part 2) – Real Python

Hugo indexes and layouts

I think that:

• Placing an _index.md in the root of the section makes it listable with a list.html template.
• Placing an index.md (no underscore!) makes that file’s content the real index of that section.

The best way to use a custom layout is to specify it explicitly in the front matter as layout: newlayout. For example for the custom list template in pages (formerly /ntb/pages), I put the layout file in ./layouts/ntb/ntblist.html and put in ./content/ntb/pages/_index.md’s front matter this:

title: "Pages"
[...]
layout: ntblist

Pycharm presentation mode and font size

Previously, I had to manually increase font sizes in Pycharm when presenting stuff in meeting, and couldn’t automate it.

Today I realized that I can change the script resolution to a lower one, giving the same results, and easily automatable through randr and a shell script!

Pycharm moving functions

“Right click -> Refactor” works not just for renaming files/folders, but also for moving functions to different files!

Miroboard moving

Holding <space> makes the mouse move the view, not the content

Logging in Python

logging — Logging facility for Python — Python 3.9.7 documentation

logger.exception() exists! Exception info is written as given by the exception handler.

Exceptions handling in Python

Was looking for a strategy to handle errors in a complex-ish applications, with logging, different levels etc.

• Three options how to deal with exceptions:¹

  • Swallow it quietly (handle it and keep running).
  • Do something like logging, but re-raise the same exception to let higher levels handle.
  • Raise a different exception instead of the original.

• Defining your custom exception¹

  class SuperError(Exception):
      def __init__(self, message):
          Exception.__init__(message)
          self.when = datetime.now()
  
  raise SuperError('Something went wrong')
  

• Re-raising the same exception after handling it ¹

  def invoke_function(func, *args, **kwargs):
      try:
          return func(*args, **kwargs)
      except Exception as e:
          print type(e)
          raise
  

• Ways to clean stuff up in try..catch blocks:²

  • try: - execute this
  • except: execute this if there’s an exception
  • else: - execute if no exceptions
  • finally: - always run this code

• Context managers

  • Alternative to finally, standard with ... syntax

• Logging best practice¹

  import logging
  logger = logging.getLogger()
  def f():
      try:
          flaky_func()
      except Exception:
          logger.exception()
          raise
  

  If you re-raise, make you sure you don’t log the same exception over and over again at different levels.¹

  The simplest way to do it is to let all exceptions propagate (unless they can be handled confidently and swallowed earlier) and then do the logging close to the top level of your application/system.¹

• Error logger decorator for the above¹

  def log_error(logger)
      def decorated(f):
          @functools.wraps(f)
          def wrapped(*args, **kwargs):
              try:
                  return f(*args, **kwargs)
              except Exception as e:
                  if logger:
                      logger.exception(e)
                  raise
          return wrapped
      return decorated
  

  And usage:

  import logging
  logger = logging.getLogger()
  
  @log_error(logger)
  def f():
      raise Exception('I am exceptional')
  

• If there are multiple decorators, that one should be the immediate next one to the function! When I did it wrong, I got an exception (ha) about “‘staticmethod’ object is not callable”.

  The correct way is:

  @staticmethod
  @return_trainer_exception(logger=None)
  

Cherry-picking commits from pycharm

Messed up merging/rebasing branches from branches from branches, but needed to merge literally a couple of commits.

So I created a clean branch from master. Then:

• Check out the target branch, the one you’re cherry-picking to
• Open the git log
• Select the commits you want to cherry-pick, right click, “cherry-pick”
• Done!

As usual, docs exist¹ and are quite readable.

PEP8 max line width of 80 characters

… is the best thing since sliced bread, I was skeptical at first but makes editing code in multiple windows so much better!

Installing python3.8 on Ubuntu 18.04 LTS

• Needed a third-party PPA² that has all the newer python versions:

    sudo add-apt-repository ppa:deadsnakes/ppa
    sudo apt install python3.8
    sudo apt install python3.8-dev
  

• Needed sudo apt-get install python3.8-venv³
• Needing to reinstall all packages for it, haha.
  • Set up locally a venv38 for this; if I source venv38/bin/activate python3 becomes python3.8 by default.
• Needed to install

python3.8-dev was added after an error I had⁴ when installing pycocotools, it didn’t find python.h when building.

Installing python locally

This describes the process well: Install python3.6+ for local user on remote machine without root access - ~/web/logs

The official documentation: 2. Using Python on Unix platforms — Python 3.9.7 documentation

Basically make altinstall is a safer version that doesn’t overwrite system-wide stuff:

make install can overwrite or masquerade the python3 binary. make altinstall is therefore recommended instead of make install since it only installs exec_prefix/bin/pythonversion.

TL;DR:

• Download the source tarball
• Then:

  ./configure --prefix=whatever
  make
  make altinstall
  

• Add the prefix to $PATH:

  export PATH=$PATH:/data/sh/tools/python3.8/bin
  

Hugo auto-reload and CSS

Just now remembered that when doing CSS stuff it’s sometimes cached, and one needs to <Shift-R> or sth similar. Hugo’s automatic reloading reloads the content/templates/…, but not the CSS!

Explains a lot of what happened the last two days.

Hugo Templating

Copypasting from the docu⁵:

• Parameters for functions are separated using spaces
• Dot-notations for methods and fields ({{ .Params.bar }})
• Things can be grouped via parentheses:
  • {{ if or (isset .Params "alt") (isset .Params "caption") }} Caption {{ end }}
• A Single Statement Can be Split over Multiple Lines:

  {{ if or 
    (isset .Params "alt") 
    (isset .Params "caption")
  }}
  

Setting directory-specific settings in vim

Given that Hugo’s markdown considers code as part of a bullet-point if it’s indented two spaces more than the *-bulletpoint’s level, and that I have a tabwidth of 4 and tabs everywhere else and two spaces were a pain…

To apply settings only within a specific directory, add this to ~/.vimrc⁶:

autocmd BufNewFile,BufRead /home/me/ntb/* set tabstop=4 softtabstop=4 shiftwidth=4 expandtab foldmethod=marker

Notably, for me it didn’t work when the path contained a symlink, had to write it explicitly.

Another option from that SO questiont was placing a ~/.vimrc in that directory⁷, allowing vim to use it by default, and sourcing the usual global one from the first line. Has security implications, may lead to issues with paths/plugins, didn’t try it.

vim tabs and spaces and indentation settings

Looking for indentation stuff for the above lead me here: Tab settings in Vim. Summary: | by Ari Sweedler | Medium

It has this description, copying verbatim:

• tabstop: display-only, how many spaces does one \t equal visually?
• shiftwidth: how many spaces does one level of indentation equal? (shifting commands, formatting, behaviour).
• softtabstop: how much whitespace to add/remove when pressing tab/backspace?
  • Disabled by default; if using tabs - we create as much whitespace as needed to get to the next tabstop
  • but when using spaces for indentation, we don’t want backspace to delete one space, then this is needed
• expandtab: should pressing <Tab> on the keyboard create spaces or a tab character?

highlight indentation levels in vim, if indentation is done with spaces

Highlighting tab-indents is easy, and I had these settings for that:

set listchars=tab:\:\ 
set listchars+=trail:◦

For spaces it’s harder.

Tried the indentLine plugin⁸, due to it using the conceal setting I couldn’t see my json-quotes and _ underscores anymore. Setting conceallevel to 1 from 2 helped only for the latter. May get fixed by colorscheme/syntax files with less concealed stuff?

Setting let g:indentLine_concealcursor = '' (by default inc) helps - text is not concealed at all in the cursor line in any of the modes. I see all concealed text and don’t see the guides. I can kinda live with that.

In any case replacing the 's in json is ugly.

Then found this excellent SO answer. set cursorcolumn cursorline highlight the entire column/row where the cursor is. Which is why I want indentation highlighting 99% of the time!

With my newfound vim knowledge, added this to ~/.vimrc:

autocmd filetype python set cursorcolumn cursorline

But this didn’t satisfy me for the dtb and I kept looking.

Then I found vim-indent-guides⁹ that changes just the background color. Settings I ended up using:

let g:indent_guides_enable_on_vim_startup = 1
let g:indent_guides_auto_colors = 0
let g:indent_guides_start_level = 2
let g:indent_guides_guide_size = 4
" autocmd VimEnter,Colorscheme * :hi IndentGuidesOdd  guibg=darkgrey  ctermbg=233
autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=233

ctermbg=233is one of the darkest black-like vim colors, there’s a nice vim colors reference¹⁰ online.

At the end, wrapped everything related to DTB and indentst in one nice startup function:

fun! SetDTB()
	set tabstop=4  shiftwidth=2 expandtab 
	foldmethod=marker
	set nocursorline nocursorcolumn 
	let g:indent_guides_auto_colors = 0
	let g:indent_guides_start_level = 1
	let g:indent_guides_guide_size = 1
	autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=236
endfu

autocmd BufNewFile,BufRead /home/me/ntb/* :call SetDTB()