serhii.net

vim open list of files from cli

vim -p `ag -l whatever`

opens each file returned by ag. (ag -l lists only the files with matches and nothing else)

Replacing jekyll-style highlight tags with standard markdown ones

In some posts I had code blocks like {% highlight html %} etc. The html/js got parsed, and some “here’s how to redirect using javascript” code got executed in the master page.

Here’s how I replaced all that syntax with the standard markdown one:

for f in `ag -l "endhighlight"`;
do cat $f | sed "s/{% highlight \(.*\) %}/\`\`\`\1/" | sed "s/{% endhighlight %}/\`\`\`/g" > $f;
done

Python dataclasses and classmethods

@dataclass
class MyClass:
  x: int = 4

@classmethod
def init_whatever(number: int)
  return cls(x=number)

Python exceptions and unittests

unittest’s self.assertRaisesRegex() is nice but couldn’t get it to work with my custom exception class.

with self.assertRaisesRegex(CustomException, "No trained model"):

It expects the message to be in e.args¹. args also gets used by the Exception class for __str__() etc, so it’s a nice thing.

Set it up easily:

class CustomException(Exception):
    def __init__(self, detailed_message: str = None):
        if detailed_message:
          self.detailed_message = detailed_message
          self.args = (self.detailed_message, )

Catching python regex exceptions

try:
  re.search("DataLoader worker(.*is killed by signal: Bus error", text)
except re.error:
  whatever()

TODO I really like this regex tutorial: Regular Expressions: Regexes in Python (Part 2) – Real Python

Hugo indexes and layouts

I think that:

• Placing an _index.md in the root of the section makes it listable with a list.html template.
• Placing an index.md (no underscore!) makes that file’s content the real index of that section.

The best way to use a custom layout is to specify it explicitly in the front matter as layout: newlayout. For example for the custom list template in pages (formerly /ntb/pages), I put the layout file in ./layouts/ntb/ntblist.html and put in ./content/ntb/pages/_index.md’s front matter this:

title: "Pages"
[...]
layout: ntblist

Pycharm presentation mode and font size

Previously, I had to manually increase font sizes in Pycharm when presenting stuff in meeting, and couldn’t automate it.

Today I realized that I can change the script resolution to a lower one, giving the same results, and easily automatable through randr and a shell script!

Pycharm moving functions

“Right click -> Refactor” works not just for renaming files/folders, but also for moving functions to different files!

Miroboard moving

Holding <space> makes the mouse move the view, not the content

Logging in Python

logging — Logging facility for Python — Python 3.9.7 documentation

logger.exception() exists! Exception info is written as given by the exception handler.

Exceptions handling in Python

Was looking for a strategy to handle errors in a complex-ish applications, with logging, different levels etc.

• Three options how to deal with exceptions:¹

  • Swallow it quietly (handle it and keep running).
  • Do something like logging, but re-raise the same exception to let higher levels handle.
  • Raise a different exception instead of the original.

• Defining your custom exception¹

  class SuperError(Exception):
      def __init__(self, message):
          Exception.__init__(message)
          self.when = datetime.now()
  
  raise SuperError('Something went wrong')
  

• Re-raising the same exception after handling it ¹

  def invoke_function(func, *args, **kwargs):
      try:
          return func(*args, **kwargs)
      except Exception as e:
          print type(e)
          raise
  

• Ways to clean stuff up in try..catch blocks:²

  • try: - execute this
  • except: execute this if there’s an exception
  • else: - execute if no exceptions
  • finally: - always run this code

• Context managers

  • Alternative to finally, standard with ... syntax

• Logging best practice¹

  import logging
  logger = logging.getLogger()
  def f():
      try:
          flaky_func()
      except Exception:
          logger.exception()
          raise
  

  If you re-raise, make you sure you don’t log the same exception over and over again at different levels.¹

  The simplest way to do it is to let all exceptions propagate (unless they can be handled confidently and swallowed earlier) and then do the logging close to the top level of your application/system.¹

• Error logger decorator for the above¹

  def log_error(logger)
      def decorated(f):
          @functools.wraps(f)
          def wrapped(*args, **kwargs):
              try:
                  return f(*args, **kwargs)
              except Exception as e:
                  if logger:
                      logger.exception(e)
                  raise
          return wrapped
      return decorated
  

  And usage:

  import logging
  logger = logging.getLogger()
  
  @log_error(logger)
  def f():
      raise Exception('I am exceptional')
  

• If there are multiple decorators, that one should be the immediate next one to the function! When I did it wrong, I got an exception (ha) about “‘staticmethod’ object is not callable”.

  The correct way is:

  @staticmethod
  @return_trainer_exception(logger=None)
  

Cherry-picking commits from pycharm

Messed up merging/rebasing branches from branches from branches, but needed to merge literally a couple of commits.

So I created a clean branch from master. Then:

• Check out the target branch, the one you’re cherry-picking to
• Open the git log
• Select the commits you want to cherry-pick, right click, “cherry-pick”
• Done!

As usual, docs exist¹ and are quite readable.

PEP8 max line width of 80 characters

… is the best thing since sliced bread, I was skeptical at first but makes editing code in multiple windows so much better!

Installing python3.8 on Ubuntu 18.04 LTS

• Needed a third-party PPA² that has all the newer python versions:

    sudo add-apt-repository ppa:deadsnakes/ppa
    sudo apt install python3.8
    sudo apt install python3.8-dev
  

• Needed sudo apt-get install python3.8-venv³
• Needing to reinstall all packages for it, haha.
  • Set up locally a venv38 for this; if I source venv38/bin/activate python3 becomes python3.8 by default.
• Needed to install

python3.8-dev was added after an error I had⁴ when installing pycocotools, it didn’t find python.h when building.

Installing python locally

This describes the process well: Install python3.6+ for local user on remote machine without root access - ~/web/logs

The official documentation: 2. Using Python on Unix platforms — Python 3.9.7 documentation

Basically make altinstall is a safer version that doesn’t overwrite system-wide stuff:

make install can overwrite or masquerade the python3 binary. make altinstall is therefore recommended instead of make install since it only installs exec_prefix/bin/pythonversion.

TL;DR:

• Download the source tarball
• Then:

  ./configure --prefix=whatever
  make
  make altinstall
  

• Add the prefix to $PATH:

  export PATH=$PATH:/data/sh/tools/python3.8/bin
  

Hugo auto-reload and CSS

Just now remembered that when doing CSS stuff it’s sometimes cached, and one needs to <Shift-R> or sth similar. Hugo’s automatic reloading reloads the content/templates/…, but not the CSS!

Explains a lot of what happened the last two days.

Hugo Templating

Copypasting from the docu⁵:

• Parameters for functions are separated using spaces
• Dot-notations for methods and fields ({{ .Params.bar }})
• Things can be grouped via parentheses:
  • {{ if or (isset .Params "alt") (isset .Params "caption") }} Caption {{ end }}
• A Single Statement Can be Split over Multiple Lines:

  {{ if or 
    (isset .Params "alt") 
    (isset .Params "caption")
  }}
  

Setting directory-specific settings in vim

Given that Hugo’s markdown considers code as part of a bullet-point if it’s indented two spaces more than the *-bulletpoint’s level, and that I have a tabwidth of 4 and tabs everywhere else and two spaces were a pain…

To apply settings only within a specific directory, add this to ~/.vimrc⁶:

autocmd BufNewFile,BufRead /home/me/ntb/* set tabstop=4 softtabstop=4 shiftwidth=4 expandtab foldmethod=marker

Notably, for me it didn’t work when the path contained a symlink, had to write it explicitly.

Another option from that SO questiont was placing a ~/.vimrc in that directory⁷, allowing vim to use it by default, and sourcing the usual global one from the first line. Has security implications, may lead to issues with paths/plugins, didn’t try it.

vim tabs and spaces and indentation settings

Looking for indentation stuff for the above lead me here: Tab settings in Vim. Summary: | by Ari Sweedler | Medium

It has this description, copying verbatim:

• tabstop: display-only, how many spaces does one \t equal visually?
• shiftwidth: how many spaces does one level of indentation equal? (shifting commands, formatting, behaviour).
• softtabstop: how much whitespace to add/remove when pressing tab/backspace?
  • Disabled by default; if using tabs - we create as much whitespace as needed to get to the next tabstop
  • but when using spaces for indentation, we don’t want backspace to delete one space, then this is needed
• expandtab: should pressing <Tab> on the keyboard create spaces or a tab character?

highlight indentation levels in vim, if indentation is done with spaces

Highlighting tab-indents is easy, and I had these settings for that:

set listchars=tab:\:\ 
set listchars+=trail:◦

For spaces it’s harder.

Tried the indentLine plugin⁸, due to it using the conceal setting I couldn’t see my json-quotes and _ underscores anymore. Setting conceallevel to 1 from 2 helped only for the latter. May get fixed by colorscheme/syntax files with less concealed stuff?

Setting let g:indentLine_concealcursor = '' (by default inc) helps - text is not concealed at all in the cursor line in any of the modes. I see all concealed text and don’t see the guides. I can kinda live with that.

In any case replacing the 's in json is ugly.

Then found this excellent SO answer. set cursorcolumn cursorline highlight the entire column/row where the cursor is. Which is why I want indentation highlighting 99% of the time!

With my newfound vim knowledge, added this to ~/.vimrc:

autocmd filetype python set cursorcolumn cursorline

But this didn’t satisfy me for the dtb and I kept looking.

Then I found vim-indent-guides⁹ that changes just the background color. Settings I ended up using:

let g:indent_guides_enable_on_vim_startup = 1
let g:indent_guides_auto_colors = 0
let g:indent_guides_start_level = 2
let g:indent_guides_guide_size = 4
" autocmd VimEnter,Colorscheme * :hi IndentGuidesOdd  guibg=darkgrey  ctermbg=233
autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=233

ctermbg=233is one of the darkest black-like vim colors, there’s a nice vim colors reference¹⁰ online.

At the end, wrapped everything related to DTB and indentst in one nice startup function:

fun! SetDTB()
	set tabstop=4  shiftwidth=2 expandtab 
	foldmethod=marker
	set nocursorline nocursorcolumn 
	let g:indent_guides_auto_colors = 0
	let g:indent_guides_start_level = 1
	let g:indent_guides_guide_size = 1
	autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=236
endfu

autocmd BufNewFile,BufRead /home/me/ntb/* :call SetDTB()

1001th post in Hugo!

Set up Hugo for DTB and partly sth I’ll call NTB, which is non-work stuff.

So far Hugo is 110/10.

Old one for now is here.

Jekyll to Hugo

TODO:

• Aliases/redirects from old posts to new ones (serhii.net/day123.html -> serhii.net/day123)
  • uglyurls: true in config does exactly this!
  • …but breaks lists/indexes somehow :(
• Look through master file for formatting issues
• Better black-background syntax highlighting if no language specified
  • Ideally make them indistinguishable from styled ones
  • And remove ghost ones like day 996
    • The problem was with my markdown syntax, apparently *I need a two space indentation from the * for it to be parsed correctly. Another reason to revisit my vim tab settings?
  • using '''text seems like a workaround:

    This is text
    No syntax highlighting
    

    And:

    This is text
    No syntax highlighting _at all_
    

• Randomized footers
• Set up Atom feed on home page
  • Or actually - I could move the entire website to Hugo, and have the index-index as a template and /dtb for the posts stuff?
• Strikethrough
  • Markdown strikethrough is ~~strikethrough~~ ¹
• Fix code listings’ width influencing the width of entire Day.

tree including hidden files

I love how intuitive it is - needed a dotfile in tree, tried tree -a, it worked.

Python unittest

setUp() and tearDown() methods in unittests get executed before/after each test method!

Unregistering Detectron2 datasets for unittests

The dictionary with the datasets is a global dictionary, which means that you can’t register_coco_instances() in separate unittests in the same file!

This worked:

if Constants.TRAIN_DATASET_NAME in MetadataCatalog.data:
    MetadataCatalog.remove(Constants.TRAIN_DATASET_NAME)
    MetadataCatalog.remove(Constants.TEST_DATASET_NAME)
    DatasetCatalog.remove(Constants.TRAIN_DATASET_NAME)
    DatasetCatalog.remove(Constants.TEST_DATASET_NAME)

Pycharm / Intellij idea visual guides for character limit

Through IDE settings one can configure whether one or multiple visual guides are shown, and the actual number of characters is configured through Settings -> Code Style.

Random

Jupyter notebooks + RISE + Reveal.js + a makefile: cornell-cs5785-2021-applied-ml/Makefile at main · kuleshov/cornell-cs5785-2021-applied-ml

TODO - Git - squashing multiple commits into one

Squash commits into one with Git - Internal Pointers (link by SO):

# Merge the last 7 commits into one
git rebase --interactive HEAD~[7]
# Merge the commits from that commit hash
git rebase --interactive 6394dc

In the latest one, the commit hash is “the hash of the commit just before the first one you want to rewrite from.”

Practically, assuming I want to squash together the a ones, I’d do git rebase --interactive B as it’s the one immediately following the ones I need.

commit a1 (latest/newest)
commit a2
commit a3
commit B
commit C

When actually doing the squashing, set squash in front of the commit lines to squash. In the next screen, leave only the commit message(s) needed.

I love how it uses vim for this! Very interesting way to do an interface.

Python PEP8 / black / flake8 / style

flake8 file.py shows issues;

black file.py applies black. black --line-length=79 file.py applies the line length as per PEP8.

Pycharm uses 119 characters as limit, coming from Intellij I think; officially PEP8 recommends 79.

German / Words

Blau sein = be drunk (heard at work)

Hugo the static site generator

My blog takes minutes to be generated, this DTB is not far from it either. I heard Hugo is fast, and I dislike most of what’s on my blog, the logical thing seems to burn it to the ground and start from zero using Hugo.

• Quick Start | Hugo
• Single posts are markdown too, with very similar font matter!
• A theme I like: Ink | Hugo Themes
• Installing themes:

cd themes
git submodule add https://github.com/chollinger93/ink-free
cd ..
echo theme = \"ink-free\" >> config.toml

• Creating a post:

hugo new posts/my-first-post.md

puts the file in ./content/posts/my-first-post.md

• Building:
  • Starting the local server: hugo server -D
  • REALLY fast, and reloaded live in my browser!
  • Building the site: hugo -D
• Configs
  • config.toml supports #comments
  • con-fig and con-tent in the same folder make my tab completion sad.
  • Configure Hugo | Hugo
  • It supports .yaml, .json and .toml configs and config directories!
• Directory structure: Directory Structure | Hugo
  • Site structure in inferred by directories: Content Sections | Hugo
    • They still need to be added to config to be in the menu
    • Nevertheless accessible when typing the URL directly
    • A subdirectory is a navigable section only if it has an _index.md
  • hugo new content/pages/one/two/test-page.md
    • The command only from parent of config
    • It generates the boilerplate, I don’t need to write a script for that! It even gets the title from the filename!
    • If there’s an archetype with the same name it’ll use that!
• Writing / content creation
  • This seems to describe everthing: Content Management | Hugo

Python typing cheatsheet & other stuff

Nice cheatsheet, not mypy-specific: Type hints cheat sheet (Python 3) — Mypy 0.910 documentation

Otherwise:

• Functions that may return None:
  • x = 1 if True else None, x would be Optional[int]
• Iterables / Sequences:
  • Iterable is anything usable inside a for
  • Sequence is anything that supports len()
  • For example:

    def f(ints: Iterable[int]) -> List[str]:
    return [str(x) for x in ints]
    

python unittests through CLI

From docu ¹:

python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method

When I’m in the directory with the test_xy.py files, running python3 -m unittest runs all of them. I can also do python3 -m unittest test_xy for that file, and python3 -m unittest test_xy.TestXY.test_specific_thing.

Debugging python from CLI through breakpoints

Found this, and it’s freaking awesome: Debugging by starting a REPL at a breakpoint is fun

Sample from there:

def make_request():
    result = requests.get("https://google.com")
    import ipdb; ipdb.set_trace()

There’s the default pdb, there’s ipdb that has to be installed.

Adding

import ipdb; ipdb.set_trace()

anywhere in the code launches a typical debug window that can be used to look into the vars etc.

Just used this for the first time to debug a python program that was running on a remote server and failing, but not locally.

SO much better than print(f"XXX {i}") and friends!

Nice tutorial about its usage: Better Python Debugging With IPDB

• n - next line in current method (=“step over”)
• s - next line of executable code anywhere (=“step into”)
• c - continue till next breakpoint
• r - continue till function returns (would be nice to learn how to do this in pycharm btw!)
• a - args - print arguments current function received
• b - adds breakpoint to locations
  • b filename.py:234
  • b <function>
  • b 123 - line in current file

Full documentation here: 26.2. pdb — The Python Debugger — Python 2.7.18 documentation

Python serializing Enums by declaring them as subclass of str

My main issue with Enum classes was that serialization is weird, especially if you’re dumping parameters. Tried again, found this: python - Serialising an Enum member to JSON - Stack Overflow

TL;DR class EnumDerivedClass(str, Enum)

import json
from enum import Enum

class LogLevel(str, Enum):
    DEBUG = 'DEBUG'
    INFO = 'INFO'

print(LogLevel.DEBUG)
print(json.dumps(LogLevel.DEBUG))
print(json.loads('"DEBUG"'))
print(LogLevel('DEBUG'))

will output

LogLevel.DEBUG
"DEBUG"
DEBUG
LogLevel.DEBUG

Google Presentations work in progress slides

“Folie überspringen” is a much better way to do what I did with setting a yellow background color - easy to see and worst case scenario it’ll just get skipped

Tensorboard and no data because wrong input folder

If you run tensorboard on a non-existing folder, you’ll get no feedback about it anywhere?.. No data on Tensorboard itself, nothing useful in CLI.

Pycharm / Intellij idea local history - for files and directories!

After some ill-fated undoing of commits, couldn’t find the work of an hour or so.

Guess what: Using Local History to Restore Code Fragments or Deleted Files | The IntelliJ IDEA Blog

I knew about local history for a file, but you can do the same for a directory, through its right-click menu in the Projects view!

Nvidia GPU/eGPU drivers blues

I already thought I had set up nvidia-smi and friends (Day 850 | Diensttagebuch (Work journal)), then didn’t use it for months, now when I tried it didn’t work anymore, nvidia-smi said “No devices found”

boltctl showed the device as connected and authorized, prime-select said nvidia was selected, modprobe showed that the correct drivers were used and dkms status had said the correct drivers were installed.

(11:53:23/10181)~/$ dkms status
nvidia, 460.73.01, 5.4.0-73-generic, x86_64: installed
nvidia, 460.73.01, 5.4.0-74-generic, x86_64: installed

(11:53:49/10182)~/$ boltctl
[snip]
 ● Lenovo ThinkPad Thunderbolt 3 Dock #2
   ├─ type:          peripheral
   ├─ name:          ThinkPad Thunderbolt 3 Dock
   ├─ vendor:        Lenovo
   ├─ uuid:          xxx
   ├─ status:        authorized
   │  ├─ domain:     domain0
   │  └─ authflags:  none
   ├─ authorized:    Mo 20 Sep 2021 09:41:16 UTC
   ├─ connected:     Mo 20 Sep 2021 09:41:16 UTC
   └─ stored:        no

 ● GIGABYTE GV-N1070IXEB-8GD
   ├─ type:          peripheral
   ├─ name:          GV-N1070IXEB-8GD
   ├─ vendor:        GIGABYTE
   ├─ uuid:          xxx
   ├─ status:        authorized
   │  ├─ domain:     domain0
   │  └─ authflags:  none
   ├─ authorized:    Mo 20 Sep 2021 09:42:35 UTC
   ├─ connected:     Mo 20 Sep 2021 09:42:35 UTC
   └─ stored:        Mo 20 Sep 2021 09:31:09 UTC
      ├─ policy:     manual
      └─ key:        no

(11:54:54/10188)~/$ lsmod
Module                  Size  Used by
nvidia_uvm           1015808  0
nvidia_drm             57344  1
nvidia_modeset       1228800  1 nvidia_drm
nvidia              34123776  17 nvidia_uvm,nvidia_modeset

(11:55:54/10192)~/$ sudo prime-select query
nvidia

What didn’t work:

• prime-select cycling to default and then back to nvidia and rebooting
• power-cycling the CPU
• Connecting it directly, not through the dock, exact same setup I had in when it was working (link above)

What worked:

• Honestly no idea
• logging into gnome, opening the driver config window, logging back into i3, rebooting?…

Offtopic, when I was googling these issues I found my own serhii.net link above on the first page of Google for the key ’“nvidia-smi “no devices were found” authorized’, which is both nice and sad at the same time :)

EDIT: the next morning it didn’t work again. None of the same magic steps in all possible orders. I think it might be an issue with the eGPU or dock or something of that level. The best way to check this would be to do the nuclear option, uninstall all drivers, and install from the beginning, but I think my monthly quota of GPU stuff is full five times over now.

Diensttagebuch / Meta

We’re on day 993 (!) of Diensttagebuch! Freaking awesome.

python pip “advanced” requirements.txt creation

Was creating a requirements.txt for detectron2, official install instructions were:

python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html

Answer specificalyl about this: python - How to format requirements.txt when package source is from specific websites? - Stack Overflow:

requirements.txt format is:

[[--option]...]
<requirement specifier> [; markers] [[--option]...]
<archive url/path>
[-e] <local project path>
[-e] <vcs project url>

<requirements specifier> is:

SomeProject
SomeProject == 1.3
SomeProject >=1.2,<2.0
SomeProject[foo, bar]
SomeProject~=1.4.2

The –option (such as the -f/–find-links) is the same as the pip install options you would use if you were doing pip install from the command line.

Therefore, in requirements.txt it ended up literally as this:

--find-links https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html detectron2

And by the way, detectron2’s own requirements.txt demonstrates nicely part of the above.

My own requirements.txt for CUDA 11.1:

opencv-python==4.2.0.32

# torch 1.9 for cuda 10.2 (for this config https://pytorch.org/get-started/locally/ has no versions in the command
# getting both exact versions from pip freeze
-f https://download.pytorch.org/whl/torch_stable.html
torch==1.9.0+cu111
torchvision==0.10.0+cu111
#torch==1.7.1
#torchvision==0.8.2

# python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html
-f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html
detectron2

grep/ag

Best part about ag is that I don’t need to escape anything with its default settings:

pip freeze | ag "(detectron|torch)"
pip freeze | grep "\(detectron\|torch\)"

pycharm test “teamcity” output bug

Suddenly stopped getting readable output. Fix is to add the env variable JB_DISABLE_BUFFERING, without any value, to the env of the test. teamcity - no output in console for unittests in pycharm 2017 - Stack Overflow