In python, when doing regex on a multiline string:

• re.MULTILINE makes ^ and $ match on each line, not just begin/end of entire string.
• re.DOTALL makes . match newline (by default it doesn’t).

Obsidian can do advanced search: Obsidian search

• “X OR y -z”
• js-flavour regex
• Special search operators for files/paths, to search on lines/blocks/sections, tasks, tags

tag: #Tag is better than searching the tag by itself, as the latter might find tags inside code listings etc etc etc

I changed the templates I use to be more repetitive but hopefully with less chances for a note meant to be private to get published on my website.

Three types of notes I want to be able to create easily:

• Diensttagebuch (public)
• Jourrnal (public)
• Personal (private)

I don’t want the Personal ones to end up left in any of the folders parsed by obyde even by chance, and if they do I don’t want them converted, and if they do - shown.

Now I just create a note, it gets put into /, I give it a name, and then run one of the three templates. The templates take care of moving it to the correct folder and prefic

Now I have three identical templates, they move the note to the correct place, prefix the file with the datetime if needed, and add boilerplate frontmatter.

Public diensttagebuch note (<C-t>), puts it into /garden/it/ and prefixes with datetime:

<% tp.file.move("garden/it/"+tp.date.now("YYMMDD-HHmm")+" "+tp.file.title) %>---
title: "<% tp.file.title %>"
tags:
  - "zc"
  - "zc/it"
  - "<% tp.file.cursor() %>"
fulldate: <% tp.date.now("YYYY-MM-DDTHH:MM:SSZZ") %>
date: <% tp.date.now("YYYY-MM-DD") %>
layout: post
hidden: false
draft: false
---

Public journal note (<C-S-t>) is pretty much identical:

<% tp.file.move("garden/rl/"+tp.date.now("YYMMDD-HHmm")+" "+tp.file.title) %>---
title: "<% tp.file.title %>"
tags:
  - "zc"
  - "zc/rl"
  - "<% tp.file.cursor() %>"
fulldate: <% tp.date.now("YYYY-MM-DDTHH:MM:SSZZ") %>
date: <% tp.date.now("YYYY-MM-DD") %>
layout: post
hidden: false
draft: false
---

Notes not meant to be published (<C-t>) get put into /Personal , but also:

• Have no date in frontmatter, obyde should loudly error out if it sees them (which it should never)
• If they magically end up published, I put literally all “don’t pulbish me” parameters in the header. Currently

Linking to blocks and headers in #obsidian

Is helpfully describeed in the autocomplete for [[:

• [[^to internal blocks (in the same page; you have to type the title by yourself, like this
• [[# for headers (Manually creating block references).
  • Can work even when linking from other pages!

Manually creating block references

When linking internally it autogenerates reference names, it seems. ^74ce58

Can I do this? ^myreference

Yes I can! Autocompletion even suggests/uses my reference!

Can _I_ do this? ^myreference

[Yes I can!](#^myreference)  Autocompletion even suggests/uses my reference!

When looking at a commit, clicking on “View the entire source for this file” symbol opens that file, and then one can navigate to folders etc as usual, they’ll all be from the current branch.

Hugo generates anchors from headers automatically ¹. Tested it - yes, except they’re lowercased and spaces get converted to - (which makes sense).

As a refresher, in HTML it’s

<h2 id="anchor">..</h2>
<a name="anchor"></a>

<a href="#anchor">anchor link </a>

One additional way to check the type hints in #python is mypy, installable as python package.

mypy -p package_name checks the typing in the package, and found some potential errors in corner cases I didn’t know about in one of the projects I’m working on!

Finds wrong typing, missing/wrong return values, that kind of stuff.

It doesn’t like what: str or Path typing output, I guess only Union[str, Path] - is there a reason for it?

In any case I like it more than Pycharm’s way of outputting things and will be using it along with black and flake8 in the future (along with typing itself).

#py/mypy

Tried to download a Teamcity artifact through wget, and apparently you can if you provide a user/pass through wget!

I assume it’s HTTP auth or something

wget --user username --password my-password https://teamcity.location/repository/download/....

Had issues, asked for help, and then learned a lot of stuff.

git rebase branchname != git rebase origin/branchname!

The first one is about the current local state of the branch, the second one about the one on remote.

BUT the one on remote as known by local != one on remote-remote! You need to update first!

git fetch --all or whatever.

I’d previouly update / pull before through PyCharm before doing that, and this abstracted all of this away from me.

To access the #obsidian console, <C-S-i> worked. It was the standard “Dev tools”.¹

Since I seem to keep forgetting: simple-scan is the program I use to talk to scanners. You can select various options (scan document, photo etc).

Keeps #scanning in the exact same PDF document until you break it.

In #pycharm, “Pin tab” exists! But then it’s not “Tab 1” etc anymore and I can’t use my shortcuts

(Those too after a long talk to MO at work, this time #py/argpars e)

Cool things about argparse:¹

• parser.add_argument('--two-words') would automatically map to args.two_words (_ vs -)!
• One can provide complex types!² For files, two options.
  • The first one allows to set file permissions etc., but it opens them and returns the handle to you, which you may not want.
  • pathlib.Path() works as expected, and even automagically parses string paths from args into the Path!
    • Additionally we can then establish that we’re working with Paths from the very beginning, getting rid of the str or Path ambiguity.

    • “Be strict and clear from the very beginning, then you don’t have to deal Path or str”

  • Sample of both from official documentation:

    parser.add_argument('a', type=argparse.FileType('w', encoding='latin-1'))
    parser.add_argument('b', type=pathlib.Path)
    

• You can get defalut values from os.environ()! Then you can also run it as

  WHATVEER_VALUE=234 python3 file.py
  

A nice structure for it all is:

1. if __name__ == '__main__': runs a function like main() getting rid of the scope issues
2. Parsing is done my a separate function, that returns the Namespace:

   def parse_args() -> argparse.Namespace:
       parser = argparse.ArgumentParser()
       parser.add_argument('--input-directory' ..)
       return parser.parse_args()
   

3. Then in main() we use it like args = parse_args(); if args.input_directory == ... This is nice also because then we don’t have to deal with an arparse object in main, just its results.

Also, in general, CLI programs have arguments like program --arg-one, not program --arg_one. I write the latter one because I still feel I’m in a python world, but Python would parse such dashed arguments into classic ones (see above). TODO look for some best practices for CLI programs, including Python ones, POSIX etc etc etc.

Really nice workshop-thing by MO at work about #py/logging

Naming loggers after the package / files

Logger names can be used to cleanly output and separate them.

Assuming one has a package with multiple files/subfolders in it, it’s possible to give each one their own logger, like this:

In the main file of the package:

logger = logging.getLogger(__package__)

In all the other files:

logger = logging.getLogger(__name__)

That way paths ./package/my_module.py lead to loggers named like package.my_module that map the semantical and the directory structure.

Changing settings of the loggers

In a setup above, one can then easily change the settings of the loggers referring to them by their names.

Configuring logging: Logging HOWTO — Python 3.10.0 documentation

Changing loglevel is easy from code,

if args.debug:
		logger.setLevel(logging.DEBUG)

logging.config allows to change the config from ini-like config files. Two main ways: logging.config.fileConfig reads ini-like config files, logging.config.dictConfig ¹ from dictionaries.

Sample .yaml that when converted to dict would change the loglevel of different loggers:

version: 1
                               
loggers:
	packageName.mymodule1:
		level: DEBUG
	packageName.mymodule2:
		level: DEBUG

These loggers can even include external ones!

Intro

Hi! Welcome to part 2 of my take on Personal Knowledge Management (PKM).

Part 1 was about things I tried in the past and why I stopped using them, along with what they taught me about my requirements for a PKM system.
Part 2 (this post) will be a description of the things I currently use.

TL; DR

Private notes

My day-to-day activities live in two very long textfiles, one for work (work.md) and one for personal stuff. The inspiration for this was this post: My productivity app is a single .txt file.

My personal notes are written almost exclusively using Obsidian, synchronized to my Android phone. I make heavy use of Obsidian Templater Plugin to pre-fill front matter, put them in its correct place, add tags, etc.

Public notes

The public part of my notes lives on my website, serhii.net, which is a static website on Hugo.

My public notes are written using Obsidian and stored as markdown. They get published on my website at Diensttagebuch - serhii.net and Journal - serhii.net respectively. The Diensttagebuch also generates a Master file, which basically concatenates all the single markdown files into a long page. I use this for quick casual searching.

All of this uses Git for version control.

I have a private VPS, where I run wallabag, that I use to quickly save links from my Android phone.

My current setup

Probably for the first time I’m very very happy with my current solution, it checks all boxes I knew I needed, and ones I had no idea I wanted or were possible.¹

Requirements from the previous post

Things I know I need from a PKM system, from the previous post:

• Data storage:
  • The data’s main location shouldn’t be a webservice. Ideally I should control both the data and the tools needed.
  • The data should live in a future-proof format, readable even without the tools
• It has to require little upkeep/maintenance
• Should be flexible and the data it gets and supports:
  • Not just links, but also pictures/files etc
  • free-form text for manual summaries of the items saved
• Taxonomy:
  • Ideally as flexible as possible, ability to add own attributes
  • Ability to use complex queries when searching
  • Quick feedback for errors or autocomplete for tags/categories
• Usability:
  • Adding stuff it should be really fast and friction-free. If it takes time and effort to add new info to it you won’t do it often.
  • Both loading times and number of steps should be very very small

Text files

Right as I was starting to work at my first real full-time job, I stumbled upon this link: My productivity app is a single .txt file

It describes using one long text file for organising stuff and notes, and appending to the bottom every day. I loved the idea.

My text files

… and decided to try it for work: most stuff I need to write down was company-specific, confidential and not fitting a wiki format, I was learning it chronologically, it was the perfect case.

Work work.md file

For example, the work file work.md:

• I start the day by pasting the date and “Today: “²
• I write my plans, copy-paste any not finished from the days before
• During the day, I append to that file. I write things like:
  • terminal command, non-trivial errors I had, how I solved them, all ungoogleable stuff
  • Log of what I worked on, to be able to easily come back to a topic or find old files
    • For example: “TICKET-1234. Downloaded dataset to $path, starting training by running $command. Eval results saved to …”
  • Notes from meetings (under the hashtag #meeting) or important conversations with people
• At the end of the day, I write the plans for tomorrow

Representative sample from work.md:

Small improvements accumulated along the way:

• Custom VIM colorscheme that:
  • highlights lines starting with TODO, XXX, done to make them easy to parse past.
  • Date block and line starting with \+ have a specific color, to easily parse headers or important stuff
• Using indentation to separate logical blocks, and then of vim folds with foldmethod=indent.
• Using special hastags/words to make search easy:
  • #meeting (formal ones), #conversation (informal), STARTED (for starting ML trainings that I’ll need to check up on later), etc.
  • Using initials for people. A line might look like this:

    Today:
    	- ...
    	- 13:00 #meeting with AB and CD about X
    ...
    #meeting with AB and CD about X:
    	Meeting minutes: http://...
    	AB wants to do Y
            - CD: we can't, why don't we ...
    	...
    

• Making it open in the same workspace on system startup

Personal / RL file

This worked really well, I soon created a second ile for real life stuff. The trigger there was seeing my 80 years old relative have such a paper notebook with daily notes (the day I came was something like “Serhii came to visit; planted cabbages”).

I use it for IBANs, recording payments, gift ideas for people, logistics, list of documents for bureaucracy. Almost everything above applies to it too.

Advantages

I really love these textfiles, and their length is the main proof:

> wc -l work.md
39538 work.md
> wc -l file.md
14063 file.md

• Work
  • Daily meetings, preparing for sprint reviews etc. is easy, I don’t have to remember what I did.
  • Solving problems is easier too: if I solve a non-trivial issue with internal or external tooling, I document it, and I can look it up next time.
  • It also works as a classic lab log, with experiments, theories, results. Some get copy-pasted into JIRA/Confluence later, but that way I can use the tools I like the most to draft them.
• Home
  • Excellent log of stuff I don’t do often but need to refer to.
  • For example, last time i needed something from IKEA 30% want available for delivery. Six months later I was ordering something else and quickly found the list of things not bought previously, by searching for “IKEA”.

Why does it work?

• Low-friction:
  • Really quick to start writing (opens automatically on i3 workspace “10”, switching to that workspace is mechanical memory now (<Win-0>))
  • I love textfiles and I love vim and it’s the lowest-friction way I know of to edit text
  • Searching is instant, including regex-based
• No-maintenance
  • Version control is automatic
  • No dependencies except a text editor
• Under my control and future-proof
  • Text is as portable and future-proof as it gets
  • Lives on my local filesystem, I can get a text editor anywhere

What’s missing

• Not part of its job description, but nevertheless:
  • Not good for saving stuff “for later”: no tags/categories, no way to search by filtering by them
  • No way to easily paste/view pictures etc.
• Not easily accessible from mobile
• Any solution to a problem I find and document stays on my hard drive. (Still better than “only in my head”, but still…)

Diensttagebuch / daily log

Live at Diensttagebuch - serhii.net.

Three things happened:

1. I saw /r/AskHistory thread, where a Dienttagebuch was mentioned as source. In English, I’d translate this as “Work journal/log/notebook”, “Lab notebook”. I thought I’d be a cool idea for a small blog, like the textfiles above but for public stuff.
2. Also I kept reading about static site generators³.
3. I also started hating Wordpress⁴ and was looking for alternatives.

Diensttagebuch basics

So for fun, I started writing a daily log of things I do.

At first I used Jekyll. I liked it. After it got too big, it started to take too much time to build (7-10 seconds). I heard Hugo was supposed to be fast, tried it out, pointing it at the same folder with makdowns as Jekyll (yay open formats), it worked, and took less than a second to do it! Very happy about the move, but of course it’s all still a permanent work in progress. ⁵

I started using it about at the same time as the textfiles, I started documenting there the things I keep having to google (how do you change the size of matplotlib plots in Jupyter?) along with the answers, and useful snippets. I also kept documenting random bits and pieces that I felt were interesting, but not worth the 20 seconds and mental effort to put into the link wiki (see part 1 of this post).

Representative day:

I wrote it in markdown from the beginning, and could easily paste syntax-highlighted code snippets, picures, clickable links.

At a certain point (after I started using Obsidian, see below) I split day123.md posts containing multiple possibly-unrelated headers (and no tags) into separate YYMMDD-HHmm-post-title.md ones. Now I could give them separate tags, and to link to them from other places! ⁶

To search for stuff, I created a so-called Master file. Initially it was a bash script that takes all the Days and dumps them into a single file (then also reused as page on Jekyll). I used both the page and the local file to quickly search for stuff (search whatever you want and skip through till you see what you’re looking for).

Advantages

1. I can quickly look for stuff, and immediately get the answer, usually without having to follow the link. ⁷
2. For some topics, easier to read/parse than the long textfile. Syntax highlighting, post titles etc. are nice.
3. It’s public:
   1. Easy to access myself from anywhere
   2. Easy to share with people. “Yes, I did some jq before. Here’s my short summary of it and links you can follow to go deeper: Day 860 - serhii.net”
4. Now it’s quite a big chunk of material. The directory with the source markdown files has 18.700 lines split into 477 files. It honestly feels good, and I hope I provided some value to other people on the internet.
   1. Sometimes people find stuff mentioned there and write me about it.
   2. Once I got an email asking to add their website to my “excellent list of $topic resources”
   3. At least three times I found my own posts on Google when looking for solutions to a problem! Usually happens with evergreen stuff like Nvidia GPU driver issues

Why does it work?

In bold differences from the textfiles above.

• Low-friction:
  • The window to write stuff is in my i3 scratchpad, accessible as <Win-minus>
  • I have templates/scripts to create a new file in the correct place, and pre-fill it with sensible values as soon as I give it a name
  • Searching is easy both from the browser and with grep
  • Deploying it is done through a shell script that commits and pushes it, builds the website, and scp-s it to serhii.net
• No-maintenance
  • Version control is automatic
  • Serving:
    • A static website is trivial to serve and wil stay available
  • Accessing:
    • I always have a text editor installed and can read markdown files
  • Building the static website:
    • Hugo is trivial to install, no database required to read the markdown files, it’s a beauty.
• Under my control and future-proof
  • Text is as portable and future-proof as it gets
  • Lives on my local filesystem, I can get a text editor anywhere

What’s missing

For quite a long time, the ability to drag-n-drop pictures into them, and the ability to edit them on mobile.

Obsidian and digital gardens

My love for Obsidian knows no bounds, but I’m using it heavily as described only for around three months (as of December 2021), much less than the two previous ones (that are more than around two years old). Not stable and mature by my standards, but if there were any deal-breakers or friction I’d have stopped long before that.

Once I went on a rabbit hole and stumbled upon two things I always loved but had no idea they were a thing, or had a name:

• Learn In Public (post publicly about your progress, create resources you wish you had found)
• Digital gardens (having a set of pages that get constantly updated as your knowledge grows instead of chronological posts)
  • You know something is a thing when it has a subreddit: /r/DigitalGardens

I heard Obsidian being mentioned often, and decided to try it out. Fell in love instantly.

Obsidian itself

Obsidian describes itself as “a powerful knowledge base on top of a local folder of plain text Markdown files”, which covers it quite well and instantly ticks the box for “data stays locally in an open format”.

I see it as a layer over markdown files that adds support for a lot of neat things without breaking them too much (still markdown, still roughly readable in whatever you use).

Desktop app

Electron-based, “just works”, dragging and dropping screenshots and it dealing with the rest feels like magic.

Android app

The main value it gives me is easy editing from my phone. Writing markdown by hand on a touchpad is painful, and the Obsidian editor does a really good job at it. And in general navigating through the pages, renaming, moving, following tags, adding templates. It’s also wonderfully configurable, and you can add its mobile toolbar the actions you want in the order you want.

But the two things that make Obsidian really powerful are templates and plugins.

Templates

They are snippets that manipulate text and variables to do some action like enter a date. With the help of Obsidian Templater Plugin, they can do much more.

211203-2305 New obsidian Templates + hotkeys for Garden (IT, RL) and personal notes - serhii.net has an example of one I use that gives an idea of what’s possible:

<% tp.file.move("garden/it/"+tp.date.now("YYMMDD-HHmm")+" "+tp.file.title) %>---
title: "<% tp.file.title %>"
tags:
  - "zc"
  - "zc/it"
  - "<% tp.file.cursor() %>"
fulldate: <% tp.date.now("YYYY-MM-DDTHH:MM:SSZZ") %>
date: <% tp.date.now("YYYY-MM-DD") %>
layout: post
hidden: false
draft: false
---

Works like this:

1. you create a new file, enter it’s name (let’s say “New file”).
2. Then you “Insert” that template. It:
   1. Moves that file to /garden/it/211206-1234 New file.md
   2. Inserts front matter, with title: being taken from the new filename
   3. Adds other frontmatter values and tags, puts your cursor at the beginning of the third.

To insert that template, you:

• on mobile, swipe down, type “ins”, get a list of matching ones; or you could have created a button for entering that template in the mobile toolbar.

  

• on desktop, <C-p> opens the command palette, see above. Or you could have defined a hotkey for that.

  

Creating that file on mobile and typing all that would have been hell. You create a template once, then it works both on desktop and the mobile app, and saves you a lot of pain.

Plugins

Obsidian has a lot of plugins, including the above mentioned templater. I use:

• esm7/obsidian-vimrc-support: A plugin for the Obsidian.md note-taking software
• pyrochlore/obsidian-tracker: A plugin tracks occurrences and numbers in your notes
  • I have some files I fill daily, basic checklists of habits I try to do daily, and I use it to read through the files in the directory, parse frontmatter, and plot the result:

This is a nice overview of some plugins: A Few of Our Favorite Obsidian Plugins – The Sweet Setup

Search

Obsidian supports quite advanced search.

It has standard AND/OR/NOT, regexes, and has special search operators to apply that to filenames, paths, tags, lines/blocks/sections, done or not done tasks. There’s also a vantage-obsidian to visually bulid complex queries.

My obsidian workflow

General

I create the files, insert the templates I wrote, add text, switch to the rendered view to easily read the results.

I add a lot of #tags and #tags/with_children and get autocomplete on them both on Linux and Android - no “will I look for this link under #coding or #programming?” anymore!

Integrating with Hugo / Diensttagebuch

I create individual posts with the template I pasted above. I use https://github.com/khalednassar/obyde to convert them into ones Hugo can parse (mostly involves updating pictures directories and stuff). I just added the line python3.8 -m obyde -c ./Scripts/obyde_config_dtb.yaml to the deploy.sh I use to deploy the rest of the blolg.

This part is still shaky and breaks down sometimes.

Personal notes

This is where its Android app especially shines, and I can edit text, task lists, templates for pages etc. I can take pictures and embed them in notes, or embed PDFs.

Last time I was flying, I just created a page where I pasted the pictures of the boarding pass QR codes and linked the obsidian page with my embedded .PDF vaccination certificate. (Usually I’d have all this as separate files in a Telegram chat with myself.)

You said you wanted self-hosted open-source solutions?

That was/is my main doubt. Obsidian is closed source, and building too much of my workflow on that feels against a lot of what I believe in. Let’s analyse.

Chances for Obsidian to survive

Obsidian has a LOT of plugins, which in my world meansa healthy and livng community. They also have paid offerings which give them some clear ways to monetize, which again gives me hope they’ll live.

What happens if it doesn’t?

Obsidian is free for personal use, and is a downloadable app, not a web application. Which means that (in theory) I don’t depend on their servers for anything, unless I use their paid Sync or Publish options. If the company gets hit by a bomb, the local applications itself should continue to work.

Data is stored in markdown, and there are multiple converters available to transform the Obsidian-specific syntax (which I don’t use often) into standard markdown, so there I’m also safe.

Still, why take the risk?

It’s just too awesome to ignore.

Advantages

Markdown, consistent and really good UX on computers and phones, ability to code templates to automate boring stuff, ability to easily sync part of the vault into a public Hugo blog? I never thought I’d see this anywhere.

Especially the android part: being able to write/edit stuff (checklists, personal notes, etc) from my computer and then continue from mobile when I have time has been really a game changer for me.

Why does it work?

• Low-friction:
  • The window to write stuff is in my i3 scratchpad, accessible as <Win-minus>
  • I wrote a lot of templates to automate away the boring stuff, and they’re quick to insert both from Android and Linux.
  • Searching can be done through Obsidian
  • Deploying it doesn’t require anything special, part of the deploy.sh script
• No-maintenance
  • The public part totally under control of the Hugo scripts
  • Locally - sync keeps working in the background
• Under my control and future-proof
  • It doesn’t export notes to markdown, it stores them in markdown. Which is “as portable and future-proof as it gets”
  • I don’t depend on their services:
    • publishing is easy to do with markdown
    • sync is easy to do through git / nextcloud / syncthing, now or in the future
    • the application runs locally, if it can’t I can use anything to edit markdown.

What’s missing

Complex taxonomies for a link database, but I don’t think I need that anymore. The Semantic MediaWiki Link wiki approach described in the previous post might have been useful earlier, but feels overblown for me now.

Obsidian search options over the markdown files that make up the public part of my notes are more than enough for me. In the time between my almost-abandonment of the link wiki and discovery of Obsidian, I’ve been happily Ctrl+F-ing through the various master files or grepping locally and that was … good enough.

That area is definitely a work in progress, we’ll see.

Wallabag for saving links

My current solution for casual “read later” bookmarking is Wallabag.

Wallabag (Save the web, freely | wallabag: a self hostable application for saving web pages) is a self-hosted solution to bookmark links. It has a really nice Android app.

It has stars, read / not_read, tags (clickable suggestions on mobile!), a “reader mode” for links, and can create RSS feeds for unread/starred/archived articles.

It has Tagging Rules, that based on variables like title / URI / reading time / … add specific tags.

When I see an interesting link on mobile and want to quickly save it, I “share to Wallabag”. Then I can access them from desktop through the web interface.

Next steps?

I want to move the links from my link wiki https://serhii.net/f/ into something else, and deactivate it. Something else will be probably a Hugo website, compatible with Obsidian.

Conclusion

It’s been a while since I wrote something long on my blog, feels awesome. (By the way, I drafted this post, partly, on mobile, through Obsidian).

Initially it was supposed to be a short summary of what I’m using now, but then I went down into a memory hole, realized how much thoughts I put into these topics all my life, and decided to write it all out. Glad I did.

I’m very happy with the Diensttagebuch approach for saving useful links/snippets. Obsidian is wonderful for personal stuff and adding posts to the blog, and I don’t miss the advanced querying options. Maybe I’ve just started to read stuff less, or stopped caring about saving it for the future.