Hugo generates anchors from headers automatically ¹. Tested it - yes, except they’re lowercased (which makes sense).

As a refresher, in HTML it’s

<h2 id="anchor">..</h2>
<a name="anchor"></a>

<a href="#anchor">anchor link </a>

One additional way to check the type hints in #python is mypy, installable as python package.

mypy -p package_name checks the typing in the package, and found some potential errors in corner cases I didn’t know about in one of the projects I’m working on!

Finds wrong typing, missing/wrong return values, that kind of stuff.

It doesn’t like what: str or Path typing output, I guess only Union[str, Path] - is there a reason for it?

In any case I like it more than Pycharm’s way of outputting things and will be using it along with black and flake8 in the future (along with typing itself).

#py/mypy

Tried to download a Teamcity artifact through wget, and apparently you can if you provide a user/pass through wget!

I assume it’s HTTP auth or something

wget --user username --password my-password https://teamcity.location/repository/download/....

Had issues, asked for help, and then learned a lot of stuff.

git rebase branchname != git rebase origin/branchname!

The first one is about the current local state of the branch, the second one about the one on remote.

BUT the one on remote as known by local != one on remote-remote! You need to update first!

git fetch --all or whatever.

I’d previouly update / pull before through PyCharm before doing that, and this abstracted all of this away from me.

To access the #obsidian console, <C-S-i> worked. It was the standard “Dev tools”.¹

Since I seem to keep forgetting: simple-scan is the program I use to talk to scanners. You can select various options (scan document, photo etc).

Keeps #scanning in the exact same PDF document until you break it.

In #pycharm, “Pin tab” exists! But then it’s not “Tab 1” etc anymore and I can’t use my shortcuts

(Those too after a long talk to MO at work, this time #py/argpars e)

Cool things about argparse:¹

• parser.add_argument('--two-words') would automatically map to args.two_words (_ vs -)!
• One can provide complex types!² For files, two options.
  • The first one allows to set file permissions etc., but it opens them and returns the handle to you, which you may not want.
  • pathlib.Path() works as expected, and even automagically parses string paths from args into the Path!
    • Additionally we can then establish that we’re working with Paths from the very beginning, getting rid of the str or Path ambiguity.

    • “Be strict and clear from the very beginning, then you don’t have to deal Path or str”

  • Sample of both from official documentation:

    parser.add_argument('a', type=argparse.FileType('w', encoding='latin-1'))
    parser.add_argument('b', type=pathlib.Path)
    

• You can get defalut values from os.environ()! Then you can also run it as

  WHATVEER_VALUE=234 python3 file.py
  

A nice structure for it all is:

1. if __name__ == '__main__': runs a function like main() getting rid of the scope issues
2. Parsing is done my a separate function, that returns the Namespace:

   def parse_args() -> argparse.Namespace:
       parser = argparse.ArgumentParser()
       parser.add_argument('--input-directory' ..)
       return parser.parse_args()
   

3. Then in main() we use it like args = parse_args(); if args.input_directory == ... This is nice also because then we don’t have to deal with an arparse object in main, just its results.

Also, in general, CLI programs have arguments like program --arg-one, not program --arg_one. I write the latter one because I still feel I’m in a python world, but Python would parse such dashed arguments into classic ones (see above). TODO look for some best practices for CLI programs, including Python ones, POSIX etc etc etc.

Really nice workshop-thing by MO at work about #py/logging

Naming loggers after the package / files

Logger names can be used to cleanly output and separate them.

Assuming one has a package with multiple files/subfolders in it, it’s possible to give each one their own logger, like this:

In the main file of the package:

logger = logging.getLogger(__package__)

In all the other files:

logger = logging.getLogger(__name__)

That way paths ./package/my_module.py lead to loggers named like package.my_module that map the semantical and the directory structure.

Changing settings of the loggers

In a setup above, one can then easily change the settings of the loggers referring to them by their names.

Configuring logging: Logging HOWTO — Python 3.10.0 documentation

Changing loglevel is easy from code,

if args.debug:
		logger.setLevel(logging.DEBUG)

logging.config allows to change the config from ini-like config files. Two main ways: logging.config.fileConfig reads ini-like config files, logging.config.dictConfig ¹ from dictionaries.

Sample .yaml that when converted to dict would change the loglevel of different loggers:

version: 1
                               
loggers:
	packageName.mymodule1:
		level: DEBUG
	packageName.mymodule2:
		level: DEBUG

These loggers can even include external ones!

Short notes about #py/poetry for package management

poetry new packagename creates a poetry project

From within the folder with the package:

• poetry install == pip3 install -r requierements.txt
• poetry shell == source .venv/bin/activate
• exit == deactivate

Basic usage | Documentation | Poetry - Python dependency management and packaging made easy:

• venvs live in {cache-dir}/virtualenvs, which on my box is /home/me/.cache/pypoetry/virtualenvs/ptest-eeSDLvcF-py3.6/bin/activate
• poetry.lock caches the resolved packages once we install things once.
  • Must mach pyproject.toml, a warning will be shown otherwise
  • It’s important to commit it to a VCS! It has the exact versions it resolves, beneficial for everyone to use them
• poetry update updates everything to the latest versions, overwriting poetry.lock
• poetry init initializes a project and creates a pyproject.toml interactively, allowing even to search for packages etc!

Adding packages:

• poetry add yaml adds a package
• poetry search yaml looks for packages in remote repos! Will tell you that you actually want pyyaml

Providing a __main__.py along with __init__.py makes the package itself executable:

$ python -m module_name

__main__.py would have an usual if __name__ == "__main__" block and run stuff imported from other files of that package.

(From MO’s python riddle at work)

Things declared in if __name__ == '__main__' are in global scope. Not because it’s special, but because ..global scope. All these bugs go away if you move main() to a separate function.

Code from SO answer:[^2]

In main:

>>> if __name__ == '__main__':
...     x = 1
... print 'x' in globals()
True

Inside a function:

>>> def foo():
...     if __name__ == '__main__':
...         bar = 1
... foo()
... print 'bar' in globals()
False

Python doesn’t have block-local scope, so any variables you use inside an if block will be added to the closest enclosing “real” scope.

Someone mentioned that if __name__ == '__main__' can happen anywhere in the code. Never thought about this

(From MO’s python riddle at work)

Things declared in if __name__ == '__main__' are in global scope. Not because it’s special, but because ..global scope.

Code from SO answer:[^2]

In main:

>>> if __name__ == '__main__':
...     x = 1
... print 'x' in globals()
True

Inside a function:

>>> def foo():
...     if __name__ == '__main__':
...         bar = 1
... foo()
... print 'bar' in globals()
False

Python doesn’t have block-local scope, so any variables you use inside an if block will be added to the closest enclosing “real” scope.

Someone mentioned that if __name__ == '__main__' can happen anywhere in the code. Never thought about this

If sync is enabled, in settings -> Sync there’s a “Deleted files” with versions and actions.

If not, unless a setting is set to delete to Obsidian’s trash, it’s left to the filesystem, so trash can or extundelete in my case or whatever.

Link by M.O.: nix-community/nix-data-science: Standard set of packages and overlays for data-scientists [maintainer=@tbenst]

Detectron’s Instances object gets created like this, creating attributes with names unknown initially:

def __init__(self, image_size: Tuple[int, int], **kwargs: Any):
    """
    Args:
        image_size (height, width): the spatial size of the image.
        kwargs: fields to add to this `Instances`.
    """
    self._image_size = image_size
    self._fields: Dict[str, Any] = {}
    for k, v in kwargs.items():
        self.set(k, v)

Which is neat.

To create an Instances object for unit tests I did:

pred_boxes = Boxes(tensor(
[
    [ 143.8892, 1166.6632, 1358.7292, 1411.6588],
    [ 131.3727,  864.3126, 1355.7804, 1144.3668],
    [ 585.6373,  747.7184,  922.6433,  815.9998]
]))
scores = tensor(
    [0.9971, 0.9967, 0.9938]
)
pred_classes = tensor([3, 3, 3])

instances = Instances(
    image_size=(2122, 1500),
    scores=scores,
    pred_classes=pred_classes,
    pred_boxes=pred_boxes
)

Found this in old markdown code from my old blog, I guess I forgot about this:

<what@ever.com>
<https://example.com>

When opening a lot of files as vim -p *.md* only 10 kept being opened, finally googled it.

Solution: adding set tabpagemax=50 to ~/.vimrc

From SO¹:

find . -name '*.php' -exec sed -i -e 's/www.fubar.com/www.fubar.ftw.com/g' {} \;

title: “211121-2123 Undoing git add / unstaging files” tags:

• “zc”
• “zc/it”
• “git” fulldate: 2021-11-21T21:11:47+0100 date: 2021-11-21 layout: post hidden: false draft: false

Two different questions here! Both options are: ¹ If you add a file for the first time, git rm --cached . or git -rm -r --cached . will reverse that.

If you want to un-add changes to a file that’s already in the repo, git reset <file> / git reset will undo that.

Obsidian

After more than a month of use, I’m still totally completely absolutely in love with Obsidian.

The standard scenario for any new technology I get excited about is something like

1. Discover something really cool
2. Read the entire Internet about it
3. Try to use it to solve all my problems, to see what sticks
4. After a week or so, either I stop using it completely, or I keep using it for a narrow use-case. I consider the latter a success.

I think this is the first time the opposite happened: I started to play with Obsidian as a way to quickly sync notes between my computer and phone (better than one-person Telegram groups or Nextcloud or Joplin or…), then started using it to write the Diensttagebuch, then realized I can drag-n-drop pictures/PDFs into it and can use it to keep my vaccination certificate and other stuff I need often on my phone, then oh nice it’s also a markdown editor, and we’re not done yet.

I’d usually avoid closed-source solutions, but it has everything I could ask for except that:

• a really active community, with blog posts being written, repositories, plugins etc being constantly developed by very different people
• notes being stored markdown, which then can be version-controlled and edited using any of my existing tools
  • this is much better than being able to “export” them to markdown, one less step to do (or forget, or easily disable). Fits seamlessly into the backup strategies being used.
• Downloadable program that works even without an Internet connection, even if Obsidian HQ gets hit by a meteorite
• Obsidian themselves having paid options¹, which means a chance, clear plan and incentive to survive, and clear ways to support them. Better than whatever I’d be able to do with an abandoned open source project I rely on. (That said, I’d love them to open source at least the client at some point.)

This mix between personal-and-not-personal stuff, and having both my phone and my laptop as first-class citizens, is something I’ve never had before, and something I like, then if I also try to make it into my published blog I’m bound to find edge cases, one of them being tags. That said until now, and including this bit, everything I tried to do worked seamlessly.

This post describes my attempt to set up tags in a way that both Obsidian’s native autocompletion/search AND Hugo’s tagging work.

Two types of tags

This blog is written in markdown and converted into a static website with Hugo. Both the Diensttagebuch and the journal take files from specific subfolders in a specific Obsidian vault, and get converted into Hugo-friendly markdown using Obyde.

There are two kinds of tags as a result of that:

• Obsidian’s #tags in the body of the text. They are the main kind supported, they get autocompleted, various #tag/subtag options can happen, etc. They get ignored by obyde/Hugo during conversion.
• Front matter tags. They look like this in yaml/obsidian:

  

  They are parsed by Obsidian, as in searching by them works, but adding them manually is painful and has no autocompletion. One needs to set the ‘metadata’ to be shown in settings for them to be displayed.
  They are the only ones understood by Obyde/Hugo, though.

I decided to write something that takes care of this for me easily.

Obsidian templates and Templater

Logical first step was Templater, which is a plugin for Obsidian I already use for my templating needs, such as generating a YAML frontmatter based only on a filename (see Day 1021).

I wanted:

• obsidian’s #tags to become part of the yaml frontmatter
• optionally - adding a line at the end of the file, like “Tags: #one #two #three” with the tags found in the frontmatter but not in the body.

I found a template² doing pretty much this in the Template Showcase. It uses purely Obsidian’s/Templater’s JS/Templating language and is very close to what I want - but not exactly, and somehow I didn’t want to write more Javascript than needed.

My solution is more of a hack, less portable, and uses Python, because that’s what I know how to write best.

My solution

Python script to parse/add the tags to frontmatter

I wrote a (quick-n-dirty) Python script that:

• gets the tags from the frontmatter of the input .md file
• gets the tags found by Obsidian from an environment variable
• finds tags found by obsidian but not in frontmatter
• rewrites the .md file with all the tags added to the frontmatter.

from frontmatter import Frontmatter
from pathlib import Path
import os
import argparse
import yaml

ENV_VAR_TAGS = 'tags'
TAGS_LINE = "\nTags: "
IGNORE_TAGS = {"zc"}
TAGS_TO_LOWER = True

def get_args() -> argparse.Namespace:
  parser = argparse.ArgumentParser(description='Remove tags')
  parser.add_argument('--input_file', type=Path, help='Input file')
  parser.add_argument('--print', default=False, action="store_true", help='if set, will print to screen instead of rewriting the input file')
  parser.add_argument('--add_new_tagline', default=False, action="store_true", help='if set, will create/edit a "Tags: " line at the end of the file with tags found in front matter but not text body')
  return parser.parse_args()


def str_tags_to_set(tags: str) -> set:
  """
  Converts tags like "#one,#two,#three" into a set
  of string tags without the "#"s:
    {'one', 'two', 'three'}
  """
  def parse_tag(tag: str):
    ret_tag = tag.lower() if TAGS_TO_LOWER else tag
    return ret_tag[1:]

  return_set = set([parse_tag(x) for x in tags.split(",")])
  return return_set

def set_tags_to_str(tags: set, ignore_tags: set = IGNORE_TAGS) -> str:
  """
  The opposite of str_tags_to_set, returns space-separated "#tag1 #tag, ..".
  Ignores tags that contain even part of any ignore_tags.
  """

  final_tags = ''
  for tag in tags:
    for i in ignore_tags:
      if i in tag:
        continue
    if TAGS_TO_LOWER:
      tag: str = tag.lower()
    final_tags+=f"#{tag} "
  return final_tags

def line_is_tag_line(line: str) -> bool:
  return line[0:len(TAGS_LINE)] == TAGS_LINE

def get_tags_in_tagline(tagline: str) -> set:
  words = tagline.split(" ")
  return {w for w in words if w[0]=="#"}

def main() -> None:
  args = get_args()
  input_file = args.input_file

  # input tags
  tags_frontmatter = set()
  tags_all = set()
  tags_obs = set()

  # output tags
  missing_obs_tags = set()
  missing_yaml_tags = set()

  parsed_yaml_fm = Frontmatter.read_file(input_file)
  frontmatter_dict = parsed_yaml_fm['attributes']
  post_body = parsed_yaml_fm['body']

  has_tags_in_fm = 'tags' in frontmatter_dict

  # all tags (yaml + #obsidian)
  env_tags_by_obsidian = os.getenv(ENV_VAR_TAGS)
  tags_all = str_tags_to_set(env_tags_by_obsidian)

  # tags in yaml frontmatter 
  tags_frontmatter = set()
  if has_tags_in_fm:
    tags_frontmatter = set(frontmatter_dict['tags'])

  # "obsidian" tags (basically #tags in the text)
  tags_obs = tags_all.difference(tags_frontmatter)

  # print(f"{input_file}: \n\
  #     all_tags: {tags_all}\n \
  #     obs_tags: {tags_obs}\n \
  #     fm_tags: {tags_frontmatter}\n"
  #     )

  # tags found in frontmatter but not in #obsidian
  missing_obs_tags = tags_frontmatter.difference(tags_obs)
  # #obsidian tags not found in frontmatter
  missing_fm_tags = tags_obs.difference(tags_frontmatter)

  if missing_fm_tags:
    if not has_tags_in_fm:
      frontmatter_dict['tags'] = list()
    frontmatter_dict['tags'].extend(missing_fm_tags)

  if missing_obs_tags and args.add_new_tagline:
    final_tags = set_tags_to_str(missing_obs_tags)

    # If last line is "Tags: "
    last_line = post_body.splitlines()[-1]
    if line_is_tag_line(last_line):
      # tags_in_tl = get_tags_in_tagline(last_line)
      # Remove last "\n" in post body
      post_body = post_body[:-1]
      # Add the missing tags to the last line
      post_body += " " + final_tags + "\n"
    else:
      # If we have no "Tags: " line, we add one
      final_string = TAGS_LINE + final_tags
      post_body+=final_string

  new_fm_as_str = "---\n" + yaml.dump(frontmatter_dict)  + "\n---"
  final_file_content = new_fm_as_str + "\n" + post_body

  # print(final_file_content)
  if args.print:
    print(final_file_content)
  else:
    input_file.write_text(final_file_content)

if __name__ == "__main__":
  main()

Obsidian templates

To use the python file, I added to Templater a “System command user function” ³ add_tags with this code:

python3 _templates/tags/py_add_tags.py --input_file "<% tp.file.path() %>" 

It calls the python file and passes to it the location of the currently edited file.

Then I create the template that calls it:

<% tp.user.add_tags({"tags": tp.file.tags}) %>

tp.file.tags returns all (all) tags found by Obsidian, so #body tags and frontmatter ones.

They get passed to the python script as an environment variable, canonical way to do this as per Templater docu⁴.

Initially I tried to pass them as parameter, but tp.file.tags passed in the system command user function always returned an empty list.

Usage

I write the text in Obsidian as usual, adding any #body tags I like, then at the end run that template through the hotkey I bound it to (<C-S-a>), done.

Better ways to do this

Problems with this solution:

• won’t work on Android
• too many moving parts

Ways to improve it:

• It’s possible to do this without Python. The template I found² uses JS to find the editor window, then the text in it, then the YAML node. Though intuitively I like parsing the YAML as YAML more.
• I wrote it with further automatization in mind, for example running it in deploy.sh so that it parses and edits all the markdown files in one go. This would mean splitting the thing by words, taking care of correctly grouping #tags-with/subtags and not looking inside code blocks etc. Not sure here. I’d still have Obsidian as ultimate source of truth about what it considers a tag and what not.

What now?

This is the first long-ish blog post I’ve written in a while. Feels awesome. Let’s see if I can get this blog started again.