Posts

    Day 1001


    Day 1000

    Python PEP8 / black / flake8 / style

    flake8 file.py shows issues;

    black file.py applies black. black --line-length=79 file.py applies the line length as per PEP8.

    Pycharm uses 119 characters as limit, coming from Intellij I think; officially PEP8 recommends 79.

    German / Words

    Blau sein = be drunk (heard at work)


    Day 997

    Hugo the static site generator

    My blog takes minutes to be generated, this DTB is not far from it either. I heard Hugo is fast, and I dislike most of what’s on my blog, the logical thing seems to burn it to the ground and start from zero using Hugo.

    • Quick Start | Hugo
    • Single posts are markdown too, with very similar font matter!
    • A theme I like: Ink | Hugo Themes
    • Installing themes:
      cd themes
      git submodule add https://github.com/chollinger93/ink-free
      cd ..
      echo theme = \"ink-free\" >> config.toml
      
    • Creating a post:
      hugo new posts/my-first-post.md
      

      puts the file in ./content/posts/my-first-post.md

    • Building:
      • Starting the local server: hugo server -D
      • REALLY fast, and reloaded live in my browser!
      • Building the site: hugo -D
    • Configs
      • config.toml supports #comments
      • con-fig and con-tent in the same folder make my tab completion sad.
      • Configure Hugo | Hugo
      • It supports .yaml, .json and .toml configs and config directories!
    • Directory structure: Directory Structure | Hugo
      • Site structure in inferred by directories: Content Sections | Hugo
        • They still need to be added to config to be in the menu
        • Nevertheless accessible when typing the URL directly
        • A subdirectory is a navigable section only if it has an _index.md
      • hugo new content/pages/one/two/test-page.md
        • The command only from parent of config
        • It generates the boilerplate, I don’t need to write a script for that! It even gets the title from the filename!
        • If there’s an archetype with the same name it’ll use that!
    • Writing / content creation

    Day 996

    Python typing cheatsheet & other stuff

    Nice cheatsheet, not mypy-specific: Type hints cheat sheet (Python 3) — Mypy 0.910 documentation

    Otherwise:

    • Functions that may return None:
      • x = 1 if True else None, x would be Optional[int]
    • Iterables / Sequences:
      • Iterable is anything usable inside a for
      • Sequence is anything that supports len()
      • For example:
          def f(ints: Iterable[int]) -> List[str]:
          return [str(x) for x in ints]
        

    python unittests through CLI

    From docu 1:

    python -m unittest test_module1 test_module2
    python -m unittest test_module.TestClass
    python -m unittest test_module.TestClass.test_method
    

    When I’m in the directory with the test_xy.py files, running python3 -m unittest runs all of them. I can also do python3 -m unittest test_xy for that file, and python3 -m unittest test_xy.TestXY.test_specific_thing.

    Debugging python from CLI through breakpoints

    Found this, and it’s freaking awesome: Debugging by starting a REPL at a breakpoint is fun

    Sample from there:

    def make_request():
        result = requests.get("https://google.com")
        import ipdb; ipdb.set_trace()
    

    There’s the default pdb, there’s ipdb that has to be installed.

    Adding

    import ipdb; ipdb.set_trace()
    

    anywhere in the code launches a typical debug window that can be used to look into the vars etc.

    Just used this for the first time to debug a python program that was running on a remote server and failing, but not locally.

    SO much better than print(f"XXX {i}") and friends!

    Nice tutorial about its usage: Better Python Debugging With IPDB

    • n - next line in current method (=”step over”)
    • s - next line of executable code anywhere (=”step into”)
    • c - continue till next breakpoint
    • r - continue till function returns (would be nice to learn how to do this in pycharm btw!)
    • a - args - print arguments current function received
    • b - adds breakpoint to locations
      • b filename.py:234
      • b <function>
      • b 123 - line in current file

    Full documentation here: 26.2. pdb — The Python Debugger — Python 2.7.18 documentation

    Python serializing Enums by declaring them as subclass of str

    My main issue with Enum classes was that serialization is weird, especially if you’re dumping parameters. Tried again, found this: python - Serialising an Enum member to JSON - Stack Overflow

    TL;DR class EnumDerivedClass(str, Enum)

    import json
    from enum import Enum
    
    class LogLevel(str, Enum):
        DEBUG = 'DEBUG'
        INFO = 'INFO'
    
    print(LogLevel.DEBUG)
    print(json.dumps(LogLevel.DEBUG))
    print(json.loads('"DEBUG"'))
    print(LogLevel('DEBUG'))
    

    will output

    LogLevel.DEBUG
    "DEBUG"
    DEBUG
    LogLevel.DEBUG
    

    Google Presentations work in progress slides

    “Folie überspringen” is a much better way to do what I did with setting a yellow background color - easy to see and worst case scenario it’ll just get skipped

    Tensorboard and no data because wrong input folder

    If you run tensorboard on a non-existing folder, you’ll get no feedback about it anywhere?.. No data on Tensorboard itself, nothing useful in CLI.


    Day 995

    Pycharm / Intellij idea local history - for files and directories!

    After some ill-fated undoing of commits, couldn’t find the work of an hour or so.

    Guess what: Using Local History to Restore Code Fragments or Deleted Files | The IntelliJ IDEA Blog

    I knew about local history for a file, but you can do the same for a directory, through its right-click menu in the Projects view!


    Day 993

    Nvidia GPU/eGPU drivers blues

    I already thought I had set up nvidia-smi and friends (Day 850 | Diensttagebuch (Work journal)), then didn’t use it for months, now when I tried it didn’t work anymore, nvidia-smi said “No devices found”

    boltctl showed the device as connected and authorized, prime-select said nvidia was selected, modprobe showed that the correct drivers were used and dkms status had said the correct drivers were installed.

    (11:53:23/10181)~/$ dkms status
    nvidia, 460.73.01, 5.4.0-73-generic, x86_64: installed
    nvidia, 460.73.01, 5.4.0-74-generic, x86_64: installed
    
    (11:53:49/10182)~/$ boltctl
    [snip]
     ● Lenovo ThinkPad Thunderbolt 3 Dock #2
       ├─ type:          peripheral
       ├─ name:          ThinkPad Thunderbolt 3 Dock
       ├─ vendor:        Lenovo
       ├─ uuid:          xxx
       ├─ status:        authorized
       │  ├─ domain:     domain0
       │  └─ authflags:  none
       ├─ authorized:    Mo 20 Sep 2021 09:41:16 UTC
       ├─ connected:     Mo 20 Sep 2021 09:41:16 UTC
       └─ stored:        no
    
     ● GIGABYTE GV-N1070IXEB-8GD
       ├─ type:          peripheral
       ├─ name:          GV-N1070IXEB-8GD
       ├─ vendor:        GIGABYTE
       ├─ uuid:          xxx
       ├─ status:        authorized
       │  ├─ domain:     domain0
       │  └─ authflags:  none
       ├─ authorized:    Mo 20 Sep 2021 09:42:35 UTC
       ├─ connected:     Mo 20 Sep 2021 09:42:35 UTC
       └─ stored:        Mo 20 Sep 2021 09:31:09 UTC
          ├─ policy:     manual
          └─ key:        no
    
    (11:54:54/10188)~/$ lsmod
    Module                  Size  Used by
    nvidia_uvm           1015808  0
    nvidia_drm             57344  1
    nvidia_modeset       1228800  1 nvidia_drm
    nvidia              34123776  17 nvidia_uvm,nvidia_modeset
    
    (11:55:54/10192)~/$ sudo prime-select query
    nvidia
    

    What didn’t work:

    • prime-select cycling to default and then back to nvidia and rebooting
    • power-cycling the CPU
    • Connecting it directly, not through the dock, exact same setup I had in when it was working (link above)

    What worked:

    • Honestly no idea
    • logging into gnome, opening the driver config window, logging back into i3, rebooting?…

    Offtopic, when I was googling these issues I found my own serhii.net link above on the first page of Google for the key ‘“nvidia-smi “no devices were found” authorized’, which is both nice and sad at the same time :)

    EDIT: the next morning it didn’t work again. None of the same magic steps in all possible orders. I think it might be an issue with the eGPU or dock or something of that level. The best way to check this would be to do the nuclear option, uninstall all drivers, and install from the beginning, but I think my monthly quota of GPU stuff is full five times over now.

    Diensttagebuch / Meta

    We’re on day 993 (!) of Diensttagebuch! Freaking awesome.

    python pip “advanced” requirements.txt creation

    Was creating a requirements.txt for detectron2, official install instructions were:

    python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html
    

    Answer specificalyl about this: python - How to format requirements.txt when package source is from specific websites? - Stack Overflow:

    requirements.txt format is:

    [[--option]...]
    <requirement specifier> [; markers] [[--option]...]
    <archive url/path>
    [-e] <local project path>
    [-e] <vcs project url>
    

    <requirements specifier> is:

    SomeProject
    SomeProject == 1.3
    SomeProject >=1.2,<2.0
    SomeProject[foo, bar]
    SomeProject~=1.4.2
    

    The –option (such as the -f/–find-links) is the same as the pip install options you would use if you were doing pip install from the command line.

    Therefore, in requirements.txt it ended up literally as this:

    --find-links https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html detectron2
    

    And by the way, detectron2’s own requirements.txt demonstrates nicely part of the above.

    My own requirements.txt for CUDA 11.1:

    opencv-python==4.2.0.32
    
    # torch 1.9 for cuda 10.2 (for this config https://pytorch.org/get-started/locally/ has no versions in the command
    # getting both exact versions from pip freeze
    -f https://download.pytorch.org/whl/torch_stable.html
    torch==1.9.0+cu111
    torchvision==0.10.0+cu111
    #torch==1.7.1
    #torchvision==0.8.2
    
    # python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html
    -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html
    detectron2
    

    grep/ag

    Best part about ag is that I don’t need to escape anything with its default settings:

    pip freeze | ag "(detectron|torch)"
    pip freeze | grep "\(detectron\|torch\)"
    

    pycharm test “teamcity” output bug

    Suddenly stopped getting readable output. Fix is to add the env variable JB_DISABLE_BUFFERING, without any value, to the env of the test. teamcity - no output in console for unittests in pycharm 2017 - Stack Overflow


    Day 989

    Detectron2 parameters train/eval/checkpoint etc

    The documentation about default confg covers all the parameters’ meanings and can be used as reference for that! detectron2.config — detectron2 0.5 documentation

    And me dreaming up cfg.MODEL.CHECKPOINT_PERIOD is exactly what they wanted to avoid by disallowing the creation of new keys.

    Highlights:

    # Number of images per batch across all machines. This is also the number
    # of training images per step (i.e. per iteration). 
    _C.SOLVER.IMS_PER_BATCH = 16
    
    

    Phone disk space & Telegram cache

    For the second time, discovered that Telegram Cache takes 40gb of disk space.

    In the phone’s own menus related to disk space, this was shown as “Pictures” taking 40gb, not the Telegram app and its cache. But this info is exposed through Telegram’s own menus.


    Day 988

    timewarrior track and :fill

    Who knew you could combine commands! This is how you start tracking tag1,tag2 starting from the end of the previous span:

    $ w track :fill tag1,tag2
    
    Backfilled to 2021-09-15T12:21:41
    Tracking "tag1,tag2"
      Started 2021-09-15T12:21:41
      Current               23:47
      Total               0:02:06
    

    Running DUC with sshfs (excluding files and filesystems)

    TL;DR:

    duc index ~/ --fs-exclude fuse.sshfs
    

    duc is about disk space, before running it the index should be built/updated. Usually similar to duc index ~/.

    If I have a sshfs mounted somewhere, the process never ends as it tries to index the folder where it’s mounted.

    Found some solutions:

    • To exclude entire filesystems, duc index ~/ --fs-exclude fuse.sshfs
      • According to the man page, this would be a comma-separated list of filesystems as found in fstab, like ext3,ext4.
      • My /etc/fstab didn’t have the sshfs filesystem, but mount called it fuse.sshfs and this worked!
    • To exclude individual files, duc index ~/ -e "*somefilename*"
      • doesn’t seem to work with folders in all variations I could think of (*folder\/file* etc).
      • So no way to exclude a folder? Except using its name and praying no other folders share it

    Bonus: -p shows progress during indexing.

    Now I have a new alias in ~/.zshrc:

    ducindex() {
    	duc index "$1" -p --fs-exclude fuse.sshfs 
    }
    

    cdd CLI alias for CD-ing to directory containing a file

    I copypaste a lot the locations of the files from pycharm/intellij to run them from CLI or something similar. Easiest way, because they are focused and I don’t need to focus on the files/project view for that. I can’t find an Action in pycharm/intellij to copypaste only the directory.

    Yet another alias for today:

    cdd() {
    	$(dirname "$1")
    }
    

    dirname gives the directory, dirname .. | cd and dirname ... | xargs cd don’t work (TODO - why?), so I’m using the zsh thing about “cd to the directory if it’s in a command by itself”.

    Now cdd /home/me/wahtever/test.py takes me to /home/me/whatever/ which will saved tens of seconds per year!

    Concatenating/splitting tiffs

    Of course tiffsplit1 has a sister tiffcp! Ubuntu Manpage: tiffcp - copy (and possibly convert) a TIFF file

    Concatenate N pages into a result.tif:

    tiffcp xaaa.tif xaab.tif xabr.tif result.tif
    

    pycharm highlights comments if they’re immediately below a TODO one and indented

    # TODO - this is highlighted yellow
    # And this is not
    
    # ... BUT!
    
    # TODO - this is highlighted yellow
    #  This too, because it's indented one space and logically belongs to the comment above!
    

    Random / vim / TODO

    I often do <CR>ddkkp or d$kp as a reverse-enter, moving what’s to the right of the cursor on the line above the current one. I’m sure something like this already exists in vim.

    Detectron2 and Yacs / YAML config / CfgNode; allow adding new keys

    Detectron’s Yacs has a github repo with documentation and examples, much better than detectron’s own: rbgirshick/yacs: YACS – Yet Another Configuration System

    This works:

    comp_cfg.set_new_allowed(True)
    comp_cfg['key'] = 'value'
    

    Interesting bit about why it’s not like this by default:

    We typically don’t use this so that typo in config file can be noticed. 2

    Additionally, this is set per leaf, not per config - you can allow adding stuff to the root but not to its existing children.

    And, still, even with comp_cfg.set_new_allowed(True), why can’t I merge_from_list etc. for non-existing keys? (TODO)

    Detectron’s logger and log.txt

    log.txt is nice and colorful on CLI, I don’t remember how to let vim interpret the CLI colors but less log.txt works magnificently.

    cfg.merge_from_file() doesn’t work with new keys · Issue #2082 · facebookresearch/detectron2


    Day 986

    Write full screen output/buffer to a file

    If you are inside a screen, and need to write the entire contents to a file (not just the ones currently visible), this will work:

    <C-a> :hardcopy -h <filename>.


    Day 974

    Random / language / English

    In the context of a raised hand in google Hangouts meeting: “Do you have a question or an opinion?” (heard at work)

    Intellij idea / Pycharm presentation mode

    …TIL at work in a remote meeting. Makes the window with the code full-screen, hides the other windows, and increases the font size. Neat!


    Day 972

    Python itertools.count()

    Yet another chapter of “python stdlib implementing most things I need better than me”, to create an infinite iterator itertools.count() is better than stuff like iter(range(100500)) (from AA’s comment in a PR)

    Read more...

    Day 968

    Detectron2, COCO datasets and negative examples

    Read more...

    Day 965

    Ethernet device ‘not managed’ in network-manager

    Read more...

    Day 961

    Pycharm Code Inspection

    Can be run on an entire folder on right click -> “Inspect Code”

    Read more...

    Day 960

    Changes in colorschemes/themes for low battery / low brightness / dark contexts

    When coding in a plane and then on a bus did some slight changes, some are useful:

    • Intellij / pycharm:
      • “Darcula” / “High contrast” themes, both for editor and for IDE, are really nice when doing stuff in the dark
        • “High contrast” especially when using low screen brightness
      • When you change the IDE theme, you get a prompt to change the editor theme too
    • kitty / CLI
      • Increased font size to 13 and made it bold - made stuff much easier to see, especially the bold part.
      • Keeping the text bold by default from now on!
        font_family      FiraCode-Bold
        font_size 12.0
        
    • Was unable to get solarized CSS files working in qutebrowser for any website I tried to
    Read more...

    Day 943

    CSS selectors based on attributes

    Was redesigning my website, looked if there’s a smarter way to color links based on whether they are internal/external than manually adding classes to them. Well there is: Attribute selectors - CSS: Cascading Style Sheets | MDN

    Read more...

    Day 942

    Telegram desktop shortcuts (especially for strikethrough text)

    Random list from the internet: Telegram Desktop Keyboard Shortcuts (hotkeys)

    Read more...

    Day 940

    Fastmail calendar

    Has nice keyboard shortcuts, viewable with ?. Heavily vim-inspired


    Day 930

    Notes about a presentation about privacy

    Deleted as they were not interesting/relevant anymore, but one of these days I’ll post my final (Russian-language) presentation somewhere here.


    Day 924

    Pycharm/intellij debugging adding watchers

    You can add things like someObject.someFunction() and basically any python code there! And it starts getting evaluated immediately after adding, even without stepping through or anything similar! This will save me a lot of “Eval code” - whose last remaining purpose can then be .. is “exploratory debugging” a thing?

    Read more...


Subscribe