Python PEP8 / black / flake8 / style

flake8 file.py shows issues;

black file.py applies black. black --line-length=79 file.py applies the line length as per PEP8.

Pycharm uses 119 characters as limit, coming from Intellij I think; officially PEP8 recommends 79.

Hugo the static site generator

My blog takes minutes to be generated, this DTB is not far from it either. I heard Hugo is fast, and I dislike most of what’s on my blog, the logical thing seems to burn it to the ground and start from zero using Hugo.

• Quick Start | Hugo
• Single posts are markdown too, with very similar font matter!
• A theme I like: Ink | Hugo Themes
• Installing themes:

  cd themes
  git submodule add https://github.com/chollinger93/ink-free
  cd ..
  echo theme = \"ink-free\" >> config.toml
  

• Creating a post:

  hugo new posts/my-first-post.md
  

  puts the file in ./content/posts/my-first-post.md

• Building:
  • Starting the local server: hugo server -D
  • REALLY fast, and reloaded live in my browser!
  • Building the site: hugo -D
• Configs
  • config.toml supports #comments
  • con-fig and con-tent in the same folder make my tab completion sad.
  • Configure Hugo | Hugo
  • It supports .yaml, .json and .toml configs and config directories!
• Directory structure: Directory Structure | Hugo
  • Site structure in inferred by directories: Content Sections | Hugo
    • They still need to be added to config to be in the menu
    • Nevertheless accessible when typing the URL directly
    • A subdirectory is a navigable section only if it has an _index.md
  • hugo new content/pages/one/two/test-page.md
    • The command only from parent of config
    • It generates the boilerplate, I don’t need to write a script for that! It even gets the title from the filename!
    • If there’s an archetype with the same name it’ll use that!
• Writing / content creation
  • This seems to describe everthing: Content Management | Hugo

Python typing cheatsheet & other stuff

Nice cheatsheet, not mypy-specific: Type hints cheat sheet (Python 3) — Mypy 0.910 documentation

Otherwise:

• Functions that may return None:
  • x = 1 if True else None, x would be Optional[int]
• Iterables / Sequences:
  • Iterable is anything usable inside a for
  • Sequence is anything that supports len()
  • For example:

      def f(ints: Iterable[int]) -> List[str]:
      return [str(x) for x in ints]
    

python unittests through CLI

From docu ¹:

python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method

When I’m in the directory with the test_xy.py files, running python3 -m unittest runs all of them. I can also do python3 -m unittest test_xy for that file, and python3 -m unittest test_xy.TestXY.test_specific_thing.

Debugging python from CLI through breakpoints

Found this, and it’s freaking awesome: Debugging by starting a REPL at a breakpoint is fun

Sample from there:

def make_request():
    result = requests.get("https://google.com")
    import ipdb; ipdb.set_trace()

There’s the default pdb, there’s ipdb that has to be installed.

Adding

import ipdb; ipdb.set_trace()

anywhere in the code launches a typical debug window that can be used to look into the vars etc.

Just used this for the first time to debug a python program that was running on a remote server and failing, but not locally.

SO much better than print(f"XXX {i}") and friends!

Nice tutorial about its usage: Better Python Debugging With IPDB

• n - next line in current method (=”step over”)
• s - next line of executable code anywhere (=”step into”)
• c - continue till next breakpoint
• r - continue till function returns (would be nice to learn how to do this in pycharm btw!)
• a - args - print arguments current function received
• b - adds breakpoint to locations
  • b filename.py:234
  • b <function>
  • b 123 - line in current file

Full documentation here: 26.2. pdb — The Python Debugger — Python 2.7.18 documentation

Python serializing Enums by declaring them as subclass of str

My main issue with Enum classes was that serialization is weird, especially if you’re dumping parameters. Tried again, found this: python - Serialising an Enum member to JSON - Stack Overflow

TL;DR class EnumDerivedClass(str, Enum)

import json
from enum import Enum

class LogLevel(str, Enum):
    DEBUG = 'DEBUG'
    INFO = 'INFO'

print(LogLevel.DEBUG)
print(json.dumps(LogLevel.DEBUG))
print(json.loads('"DEBUG"'))
print(LogLevel('DEBUG'))

will output

LogLevel.DEBUG
"DEBUG"
DEBUG
LogLevel.DEBUG

Google Presentations work in progress slides

“Folie überspringen” is a much better way to do what I did with setting a yellow background color - easy to see and worst case scenario it’ll just get skipped

Tensorboard and no data because wrong input folder

If you run tensorboard on a non-existing folder, you’ll get no feedback about it anywhere?.. No data on Tensorboard itself, nothing useful in CLI.

Pycharm / Intellij idea local history - for files and directories!

After some ill-fated undoing of commits, couldn’t find the work of an hour or so.

Guess what: Using Local History to Restore Code Fragments or Deleted Files | The IntelliJ IDEA Blog

I knew about local history for a file, but you can do the same for a directory, through its right-click menu in the Projects view!

Nvidia GPU/eGPU drivers blues

I already thought I had set up nvidia-smi and friends (Day 850 | Diensttagebuch (Work journal)), then didn’t use it for months, now when I tried it didn’t work anymore, nvidia-smi said “No devices found”

boltctl showed the device as connected and authorized, prime-select said nvidia was selected, modprobe showed that the correct drivers were used and dkms status had said the correct drivers were installed.

(11:53:23/10181)~/$ dkms status
nvidia, 460.73.01, 5.4.0-73-generic, x86_64: installed
nvidia, 460.73.01, 5.4.0-74-generic, x86_64: installed

(11:53:49/10182)~/$ boltctl
[snip]
 ● Lenovo ThinkPad Thunderbolt 3 Dock #2
   ├─ type:          peripheral
   ├─ name:          ThinkPad Thunderbolt 3 Dock
   ├─ vendor:        Lenovo
   ├─ uuid:          xxx
   ├─ status:        authorized
   │  ├─ domain:     domain0
   │  └─ authflags:  none
   ├─ authorized:    Mo 20 Sep 2021 09:41:16 UTC
   ├─ connected:     Mo 20 Sep 2021 09:41:16 UTC
   └─ stored:        no

 ● GIGABYTE GV-N1070IXEB-8GD
   ├─ type:          peripheral
   ├─ name:          GV-N1070IXEB-8GD
   ├─ vendor:        GIGABYTE
   ├─ uuid:          xxx
   ├─ status:        authorized
   │  ├─ domain:     domain0
   │  └─ authflags:  none
   ├─ authorized:    Mo 20 Sep 2021 09:42:35 UTC
   ├─ connected:     Mo 20 Sep 2021 09:42:35 UTC
   └─ stored:        Mo 20 Sep 2021 09:31:09 UTC
      ├─ policy:     manual
      └─ key:        no

(11:54:54/10188)~/$ lsmod
Module                  Size  Used by
nvidia_uvm           1015808  0
nvidia_drm             57344  1
nvidia_modeset       1228800  1 nvidia_drm
nvidia              34123776  17 nvidia_uvm,nvidia_modeset

(11:55:54/10192)~/$ sudo prime-select query
nvidia

What didn’t work:

• prime-select cycling to default and then back to nvidia and rebooting
• power-cycling the CPU
• Connecting it directly, not through the dock, exact same setup I had in when it was working (link above)

What worked:

• Honestly no idea
• logging into gnome, opening the driver config window, logging back into i3, rebooting?…

Offtopic, when I was googling these issues I found my own serhii.net link above on the first page of Google for the key ‘“nvidia-smi “no devices were found” authorized’, which is both nice and sad at the same time :)

EDIT: the next morning it didn’t work again. None of the same magic steps in all possible orders. I think it might be an issue with the eGPU or dock or something of that level. The best way to check this would be to do the nuclear option, uninstall all drivers, and install from the beginning, but I think my monthly quota of GPU stuff is full five times over now.

Diensttagebuch / Meta

We’re on day 993 (!) of Diensttagebuch! Freaking awesome.

python pip “advanced” requirements.txt creation

Was creating a requirements.txt for detectron2, official install instructions were:

python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html

Answer specificalyl about this: python - How to format requirements.txt when package source is from specific websites? - Stack Overflow:

requirements.txt format is:

[[--option]...]
<requirement specifier> [; markers] [[--option]...]
<archive url/path>
[-e] <local project path>
[-e] <vcs project url>

<requirements specifier> is:

SomeProject
SomeProject == 1.3
SomeProject >=1.2,<2.0
SomeProject[foo, bar]
SomeProject~=1.4.2

The –option (such as the -f/–find-links) is the same as the pip install options you would use if you were doing pip install from the command line.

Therefore, in requirements.txt it ended up literally as this:

--find-links https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html detectron2

And by the way, detectron2’s own requirements.txt demonstrates nicely part of the above.

My own requirements.txt for CUDA 11.1:

opencv-python==4.2.0.32

# torch 1.9 for cuda 10.2 (for this config https://pytorch.org/get-started/locally/ has no versions in the command
# getting both exact versions from pip freeze
-f https://download.pytorch.org/whl/torch_stable.html
torch==1.9.0+cu111
torchvision==0.10.0+cu111
#torch==1.7.1
#torchvision==0.8.2

# python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html
-f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html
detectron2

grep/ag

Best part about ag is that I don’t need to escape anything with its default settings:

pip freeze | ag "(detectron|torch)"
pip freeze | grep "\(detectron\|torch\)"

pycharm test “teamcity” output bug

Suddenly stopped getting readable output. Fix is to add the env variable JB_DISABLE_BUFFERING, without any value, to the env of the test. teamcity - no output in console for unittests in pycharm 2017 - Stack Overflow

Detectron2 parameters train/eval/checkpoint etc

The documentation about default confg covers all the parameters’ meanings and can be used as reference for that! detectron2.config — detectron2 0.5 documentation

And me dreaming up cfg.MODEL.CHECKPOINT_PERIOD is exactly what they wanted to avoid by disallowing the creation of new keys.

Highlights:

# Number of images per batch across all machines. This is also the number
# of training images per step (i.e. per iteration). 
_C.SOLVER.IMS_PER_BATCH = 16

Phone disk space & Telegram cache

For the second time, discovered that Telegram Cache takes 40gb of disk space.

In the phone’s own menus related to disk space, this was shown as “Pictures” taking 40gb, not the Telegram app and its cache. But this info is exposed through Telegram’s own menus.