serhii.net

In the middle of the desert you can say anything you want

Master file

pytest fixture to make pytest-datafiles return a pathlib.Path

pytest-datafiles · PyPI is nice but returns a py.path instead of pathlib.Path.

Tried to write something to make it convert automatically.

ASSETS_DIR = Path(__file__).parent / "assets"

@pytest.fixture
def pfiles(datafiles):
    # Fixture that converts pytest-datafiles' py.path into a pathlib.Path
    return Path(str(datafiles))

@pytest.mark.datafiles(PROJ_DIR)
def test_read_meta_json(pfiles):
	assert do_sth_with_file(pfiles)

First nontrivial fixture I write, maybe a really bad idea to do it like that. This feels like a general use case and someone had to have had this problem

pytest use conftest.py to run python code before the tests

A conftest.py file gets imported and run before all the other ones.

Pytest resolves all imports at the very beginning, I used conftest.py it to import a package so that it’ll be the one used by the imports in files that are imported in the tests (seeing that there’s a mypackage already imported, subsequent import mypackages are ignored)

(Can I think of this as something similar to an __init__.py?)

Using pytest-datafiles for assets in pytest

pytest-datafiles · PyPI allows copying files to a temporary directory, then they can be modified etc. Really neat!

Sample:

ASSETS_DIR = Path(__file__).parent / "assets"
PROJ_DIR = ASSETS_DIR / "project_dir"

konfdir =  pytest.mark.datafiles(PROJ_DIR)

@konfdir
def test_basedir_validity(datafiles):
	assert directory_is_valid(datafiles)

Also love this bit:

Note about maintenance: This project is maintained and bug reports or pull requests will be addressed. There is little activity because it simply works and no changes are required.

SADLY this means that returned path is py.path, I’m not the only one complaining about that1

Pytest has newer native fixtures that use Pathlib (Temporary directories and files — pytest documentation) but datafiles hasn’t been moved to them.


  1. py.path vs pathlib.Path · Issue #7 · omarkohl/pytest-datafiles ↩︎

Easier python logging setup with argparse's 'dest' parameter

I find this approach1 brilliant (and of course it works with everything split in separate functions a la my last post: 211124-1744 argparse notes):

import argparse
import logging

parser = argparse.ArgumentParser()
parser.add_argument(
    '-d', '--debug',
    help="Print lots of debugging statements",
    action="store_const", dest="loglevel", const=logging.DEBUG,
    default=logging.WARNING,
)
parser.add_argument(
    '-v', '--verbose',
    help="Be verbose",
    action="store_const", dest="loglevel", const=logging.INFO,
)
args = parser.parse_args()    
logging.basicConfig(level=args.loglevel)

And TIL about dest= that will make my life much easier too by outsourcing more logic to argparse.


  1. python - Easier way to enable verbose logging - Stack Overflow ↩︎

Git and execution of shell commands

Today, I ran this:

git commit -m "TICKETNAME Export of X generated with `name-of-some-utility`"

Commit message on gitlab was

"TICKETNAME Export of X generated with (Starting the export of data, wait till it downloads...)"

Clear but fascinating way it can break.


Do I want to get a clear picture of all the various levels of escaping, including globs, backticks, backslashes etc. happening in the shell?

Why doesn’t the # in git commit -m "Ticket #1231" result in a string with the 1234 commented out and a syntax error? I know it doesn’t but I wouldn’t be able to predict that behaviour without this knowledge. Would single quotes change much? How to actually comment the rest of the line this way?

What are the rules that decide whether a * gets expanded by the shell or passed to, say, scp as-is? Etc. etc. etc.

It’s all knowable and learnable, but I was never sure whether the ROI was worth it for me. Till now trial and error always worked in the rare instances I have to do something complex with bash scripts, but this is the first time it bites me in real life in an unexpected way.

Python package import patterns link + __init__ stuff

This looks really interesting! It’s not about the syntax, but about the basic design philosophies + examples of packages that use it.

What’s init for me? Designing for Python package imports | Towards Data Science

Other stuff I learned about __init__.py:

  • You can use it to enforce import order 1
  • You can use it to declare package variables
  • Automatically import modules from a package2

Stuff I discovered:

  • You can set a breakpoint in pdb physically into an __init__.py, and for example look at the stack of what called it with w

  1. What is init.py? and what should I put in it? ↩︎

  2. Package Initialization – Real Python ↩︎

Changing screen brightness on linux, on hardware and software level

Connected an external screen, it was dark, googled for a solution after resetting redshift settings didn’t work.

So, there are a lot of ways to change brightness (SO1).

xbacklight works with hardware-level brightness for the devices that support it.

For the others, software-level changing of gamma values is what’s usually needed, and what I did with a lot of different programs before. This worked this time:

xrandr --output LVDS1 --brightness 0.5

(As a bonus, it uses the already well-know and well-loved xrandr.)

Sad that arandr can’t do brightness though, but there are reasons (missing –brightness features (#35) · Issues · arandr / ARandR · GitLab)

From there I learned that ddcondrol is the way to change brightness for external monitors on hardware level, and that Jolmberg/wmbright: Monitor brightness control dockapp is a back-end that tries to do everything.


  1. How to change LCD brightness from command line (or via script)? - Ask Ubuntu ↩︎

poetry pytest takes too long to collect + tell it to ignore certain directories

pytest took seconds at the “Collecting…” stage.

I had a directory with a lot of tiny files (./data_1234/) in the poetry package folder, and blamed it initially.

SO1 told me that the syntax to ignore a folder is

[tool:pytest]
norecursedirs = subpath/*

Wildcards are nice and data*/* was the first attempt.

Nothing.

Then I without success tried this:

testpaths="tests"

After a one-hour saga, I found that the culprit was a package that I was using. The tests imported my package, which imported the slow package, and it takes seconds to do so.

‘Collecting’ seems not to be only “find test files”, but it reads them and imports them and all their dependencies.

Waiting time went back to normal as soon as I commented out importing my package from the test.


  1. python - How to tell py.test to skip certain directories? - Stack Overflow ↩︎

gitlab creating branch from Issue

From within an issue, use the dropdown left of “Create merge request” -> Create branch, will create a branch with the format “issue_n-issue_title”, for example 3-this-is-issue-number-three.

Order of directories inside a python project

If you use a directory structure like this:

resources/
src/project_name/
tests/
[...]

then you get these directories in the same order regardless of the name of the project! Then it’s always uniform, muscle memory has a chance, etc.

python pdb stops on keyboard interrupt

<Ctrl-C> of a program running inside pdb (python3 -m pdb myscript.py or whatever) doesn’t kill the program, but drops you in the debugger!

Useful when you suspect there’s an infinite loop somewhere, and want to see what exactly is the program doing when it starts using 120% of your CPU

installing noisetorch on Mint with permissions and setuid and CAP_SYS_RESOURCE

Installed noisetorch, it complained about CAP_SYS_RESOURCE like the last time and I fixed it by installing polkit like the last time, didn’t work though.

Issue seems to be that by default Mint has the home partition mounted with nosetuid1, confirmed by doing mount.

Fix was to put the binary in /opt, the prompt is the same but after entering the password it works and I see the expected interface.


  1. is a data partition best mounted with NOEXEC & NOSUID? - Linux Mint Forums ↩︎

vnstat for monitoring traffic

Use-case - using limited mobile internet.

vnstat is nice. sudo apt install vnstat, service has to be started/enabled through systemctl as usual.

Logs traffic with 5-minute granularity, so for the first 5 minutes after install will say that there’s not enough information :)

vnstat -5 returns the last hours in 5-minute interval, -h/-d/-m is hourly/daily/monthly.

-i selects the interface (otherwise all existing non-zero ones will be shown).

pdppp instead of pdb and ipdb for python debugging

pdbpp is a drop-in replacement for pdb, and I like it more than ipdb for some reason.

Installing it makes it the default one imported when importing pdb (incl. by pytest, python’s breakpoint() etc!)

Really nice tutorial: pdb++, a drop-in replacement for pdb (the Python debugger) | PythonRepo

Vanilla-pdb cheatcheet: Python Debugger Cheat Sheet - Kapeli

Features not present in pdb that I love:

  • ll outputs the text of the current function
  • sticky updates the function listing with each new line, giving a nice interactive visual feeling to the debugging process

pytest -s works to make it play nice with the stdouts generated by pdbpp.

Python expanding a list by assigning multiple elements to a slice

Saw this in the python pandoc cookbook1

holder[index:index+1] = split_home(elt)

Wow.

Never thought I could assign multiple elements to a slice!


  1. Cookbook - Pandoc (Python) ↩︎

First use of python 3.8 walrus operator!

Wooho!

files = list(input_dir.glob("*.md"))[: cs.limit]
if output_path.is_file() and ((l := len(files)) != 1):
    raise ValueError(f"Can't write {l} files to single file {output_dir}")

Had to use additional parentheses around the actual assignment. Without that, black fails in an interesting way:

error: cannot format smw_to_hugo/yaml_converter.py: cannot use --safe with this file; failed to parse source file.

kitty terminal size issues

Had weird issues with kitty terminal output being wrong, lines in vim/nvim being in wrong places, usually because it thought the terminal was a different size than it really was (blamed it on nvim initally, but the problem happened in other complex CLI programs too, notably tig).

$TERMINFO wasn’t set, and the terminfo file was nowhere to be found. The package kitty-terminfo was installed though.

In any case, downloaded the terminfo file from the repo and set the env variable manually in zshrc, now it works:

export TERMINFO="$HOME/.config/kitty/xterm-kitty"

python None in slice notation

After for the nth time writing awkward code like

if limit is None: 
    limit = len(mylist)

decided to see if there’s a better way. Looked into the walrus operator etc,but decided to test what I get with None.

Well, mylist[:None] works! No errors, I’d guess I get a copy of it same as mylist[:].

Will save me hundreds of lines in the future!

Docu about slice1 is terse, says it uses range(start,end,step) under the hood with start and step defaulting to None. But range doesn’t accept None for all arguments! TODO for later I guess.


  1. Built-in Functions — Python 3.10.1 documentation ↩︎

representing empty strings in ini files

In the context of reading a settings.ini from python’s decouple1 config lib, this works as empty string

YAML_CONVERTER_PREFIX=

has to be cast to string though:

D_YAML_CONVERTER_PREFIX = config("YAML_CONVERTER_PREFIX", cast=str)

These don’t, these are strings containing two characters, "" and '' respectively.

YAML_CONVERTER_PREFIX=""
YAML_CONVERTER_PREFIX=''

  1. python-decouple · PyPI ↩︎

vim automatically use the last search in search and replace

Just discovered this! In vim, if I skip the pattern, it’ll take the one last searched for:

/mypattern
:s//newpattern/g

mypy disabling individual warnings

Things I can pass to mypy like mypy --disallow-any-generics can be configured in pyproject.toml:

[tool.mypy]
show_error_codes = true
warn_unused_ignores = false
disallow_any_generics = false
ignore_missing_imports = true

nvim

Is nice! It transparently got all vim’s configs plugins and they seems to work!

set runtimepath^=~/.vim runtimepath+=~/.vim/after
let &packpath = &runtimepath
source ~/.vimrc

A Complete Guide to Neovim Configuration for Python Development - jdhao’s blog

jedi-vim and deoplete

deoplete for faster completions, jedi-vim for goto and friends.

davidhalter/jedi-vim: Using the jedi autocompletion library for VIM.

Interesting bindings:

let g:jedi#usages_command = "<leader>n"
let g:jedi#goto_command = "<leader>d"
let g:jedi#rename_command = "<leader>r"
let g:jedi#documentation_command = "K"

But it didn’t work for packages not living inside the default python environment, and manually each venv would be tedious. poet-v to the rescue!

let g:poetv_executables = ['poetry']
map <leader>va :PoetvActivate<CR>

Deoplete1 is an autocomplete framework (nvim-only, was my last reason for switching), deoplete-jedi2 makes it use jedi.

To select on enter, had to add this to vimrc/nvimrc:

set completeopt+=noinsert

In general deoplete faq in vim help is much longer than the one on their github repo.

nvie/vim-flake8: Flake8 plugin for Vim, <F7> to run it on the current buffer.


  1. Shougo/deoplete.nvim: Dark powered asynchronous completion framework for neovim/Vim8 ↩︎

  2. deoplete-plugins/deoplete-jedi: deoplete.nvim source for Python ↩︎

Python best practices for 2021

Python Best Practices for a New Project in 2021 - Alex Mitelman

Describes a setup that uses poetry, black, flake8, pytest, mypy and new to me isort to sort imports.

The Fast track section has a TL;DR of how to create that setup.

I also really like this intro to poetry: Package Python Projects the Proper Way with Poetry

Python click getting default values from config file

Found a post1 about it.

But I like much more Click’s way to do this (Options — Click Documentation (8.0.x)):

@click.option(
    "--username", prompt=True,
    default=lambda: os.environ.get("USER", "")
)

Of course, os.get.environ can be replaced by python-decouple’s config().

Lastly, ini files support interpolation2 (%(whatever)s)! Final solution:

[settings]
EXPORT=../../exports
CATS_INPUT=%(EXPORT)s/cats.json
@click.option(
    "--input-file",
    "-i",
    type=click.Path(exists=True, path_type=Path),
    default=lambda: config("CATS_INPUT"),
)

Also TIL if I use quotes in the ini file, they’ll become part of the final filename.


  1. Knowledge Bits — Setting Default Option Values from Config Files with Click ↩︎

  2. configparser — Configuration file parser — Python 3.10.1 documentation ↩︎

Python dotenv and python-decouple to separate configs from code

Stumbled upon python-decouple · PyPI, which seems like a “better” dotenv (supports casting, defaults etc)

For example, this is a settings.ini in poetry project root:

[settings]
ECHO=True

I can overwrite these parameters like ECHO=False poetry run python myscript.py

Neat!

Blues in setting qutebrowser as default browser

xdg-settings gets the award for least intuitive interface ever.

  • xdg-settings get default-web-browser was firefox.
  • xdg-settings set default-web-browser qutebrowser.desktop is quiet
  • xdg-settings get default-web-browser is still firefox.
  • man page says that the errors are shown as …return code??
  • echo $? returned 2, which is file not found basically.
  • Bonus points for not accepting -h (only --help), and having --list as a parameter, but get/set as commands.
> xdg-settings set default-web-browser

xdg-settings: invalid application name

oh well.

Making a script into an application

For an executable (..qutebrowser.sh) to be an ‘application’, it has to have a .desktop file in ~/.local/share .1

For qutebrowser, created this:

[Desktop Entry]
Name=Qutebrowser
Comment=Qutebrowser
Exec="~/.local/bin/qb %f"
Terminal=true
Type=Application
StartupNotify=true
MimeType=application/x-www-browser;
Keywords=python;
  • To test it, desktop-file-validate qutebrowser.desktop
  • To refresh the db, sudo update-desktop-database
  • sudo desktop-file-install qutebrowser.desktop then put it in /usr/share/applications 2

This describes all the things needed to set qb as default browser: New option for open link with browser · Issue #716 · RocketChat/Rocket.Chat.Electron

At the end, symlinked /usr/bin/qb to it’s location in my home folder, maybe the universe will come crashing on me but then I don’t have to mess with the usual creation of bash runner scripts in ~/.local/bin/.. to have it globally available. Including for things like update-alternatives that seem to want a global thing.


  1. [ Main docu for this is UnityLaunchersAndDesktopFiles - Community Help Wiki↩︎

  2. (learned it when it failed because of no sudo↩︎

Obsidian illegal names don't allow sync

Created a file with -> in the name, it didn’t appear on mobile, checked sync logs - not there because the name is “illegal”. Is not allowing > a global thing or only for Android?

Exporting Pycharm settings

To Export settings, File -> Manage IDE Settings -> Export Settings 1

Interestingly the first google result was the similarly named Share your IDE settings | PyCharm, which is a feature in Pycharm Professional and is closer to syncing than to exporting.


  1. Share PyCharm Settings | PyCharm ↩︎

Port forwarding through ssh config

  • ssh -L 6006:127.0.0.1:6006 servername -p 1234 maps port 6006 of servername to localhost:6006, using ssh that’s running there on port 1234
  • multiple ports are possible by passing multiple -L arguments

If you do it often, you can add these settings to ~/.ssh/config:

 Host pf
 Hostname servername
 LocalForward 6007 localhost:6007
 LocalForward 6006 localhost:6006
 Port 1234   

…and then you connect to it as ssh pf.

Screen tips

  • Screen screens:
    • screen -R screename attaches a screen with this name or creates it.
      • Tab completion works!
      • You can only write part of the name, will work if it’s enough to identify it
    • <C-a> :sessionname newscreenname renames an existing instance
  • ~/.screenrc exists. Some useful settings:
    • defscrollback 200000 for “infinite” scrollback
    • deflog on to log everything automatically
  • Using screen when no screen is installed1 : connect to it with ssh from any other server that does have screen installed.

  1. thought of this myself and really proud of it ↩︎

sshfs configs

sshfs mounts a remote folder to one on the local filesystem.

  • sshfs server:/data/me ./local-folder -p 12345
  • sshfs -o Ciphers=aes128-ctr -o Compression=no server:/data/me ./local-folder -p 12345 may be faster

When I tried it at the beginning it was horribly slow, the problem was the zsh prompt that had info about the current git repo. Disabling it or using bash solved the issue.

When backing stuff up, check if there are any symlinks!

If you copy a directory, there may be symlinks there, that will also show fine when you tree or cat or whatever. What saved me was their different color in the terminal.

.. How did people do this in b/w terminals?

TODO How can I avoid this in the future, given my heavy symlinks use?

Inverted index

An Inverted index - Wikipedia is a mapping from content to its location/name, as opposed to the usual case of name-to-content. One use is searching.

IPDB move through individual frames

Pressing u / d moves you through the individual frames of the stack.

Also TODO look into using it to run stuff and debug automatically on fail, without editing the source code.1


  1. Better Python Debugging With IPDB ↩︎

IPython

Stumbled yet again1 on mentions of IPython and decided to look into it, prev. assumption being that it’s the same or almost the same thing as Jupyter Notebook. (Also the i in ipdb stands for IPython-enabled, apparently).

It’s not., it’s a separate interactive superset of the Python cli that’s runnable by itself through python3 -m IPython.

Which in turn feels like a better / more interactive shell that can also do magic commands (%xxx) that I’ve seen in Google Colab / Jupyter; additionally understands bash stuff as-is and does other cool stuff. Definitely worth looking into.

ALSO the same article1 mentions a way of using IPython inside ipdb, quoting:

ipdb> from IPython import embed
ipdb> embed() # drop into an IPython session.
        # Any variables you define or modify here
        # will not affect program execution

To run a program with ipdb without editing the source and dropping in an ipdb prompt when if it breaks from shell:

python3 -m ipdb script.py

Took another look at the official docu 26.2. pdb — The Python Debugger — Python 2.7.18 documentation:

  • p prints the expression following, pp pretty-prints it.

  1. Better Python Debugging With IPDB ↩︎

pycharm already running fix

Pycharm froze, killed it with killall I think, didn’t see it in the process list even (ps aux | grep pycharm) but couldn’t start it either because it detected an already running instance and refused to start.

The Internet1 suggested pkill -f pycharm killed whatever was remaining, and I could start it after that. Still no idea what happened though.


  1. https://stackoverflow.com/questions/68449482/pycharm-is-already-running-while-trying-to-launch-from-ubuntu-terminal ↩︎

Python Union typing

In Python 3.10+, Unions (Union[str, Path]) can be also written as str | Path1

… And the syntax str or Path I’ve been using and getting no errors from, apparently, doesn’t exist at all. TODO - why did it work?


  1. Built-in Types — Python 3.10.1 documentation ↩︎

Git sparse checkout

Had a big repo, wanted to clone only some folders.

The setup below automatically fetched the subfolders I added to the sparse-checkout set.

git clone --filter=blob:none --no-checkout --branch main ssh://git@me.me/my/repo.git
cd myrepo
git sparse-checkout init --cone
git sparse-checkout set "activitywatch" "friends" ".task" ".timewarrior"

Options for adding search to Hugo

https://gohugo.io/tools/search/

It boils down to creating an index (json) then using something to search in it client side

Once an index is built, Lunr seems the way to do with this: https://lunrjs.com/docs/lunr.Query.html#~Clause

It seems flexible enough, including ability to search inside taxonomies.

python import this

import this

A coworker reminded be of this gem; quoting him:

The order is important. My favourite one is ‘explicit is better than implciit’

Python pytest workshop part 2

Recap

This is part two of 211209-1354 Python testing basics with poetry and pytest. Fixtures scopes work similarly to the various setup/teardown functions of unittest, can be per module/class/etc.

Failure

Expecting a test to fail

@pytest.mark.xfail(reason="Reason why it's supposed to fail")
def test_...

Expecting a test to raise an exception

For a specific exception, you assert that it raises that exception type and then can do asserts on the exception that is raised.

def test_whatever():
  with pytest.raises(Exception) as excinfo:
    raise Exception("oh no")
  assert str(excinfo.value) == "oh no"

Regex also works (example directly from pytest.raises() API Reference

>>> with pytest.raises(ValueError, match=r'must be \d+$'):
...     raise ValueError("value must be 42")

## Services (skipped, see below)
### Creating fixtures that get used automatically
```python
@pytest.fixture(autouse=True)
def skip_servicetest(request, run_services):
  if request....
    pytest.skip("skipped because X")

Using the filesystem

pyfakefs creates a fake filesystem that gets used transparently.

from pyfakefs.fake_filesystem import FakeFilesystem

@pytest.fixture
def common_fs(fs: FakeFilesystem):
  fs.create_dir(Path("/tmp/common"))
  fs.create_file("/tmp/common")

def test_filesystem_fixture(common_filesystem):
  assert os.path.exists("/tmp/common")
  assert os.path.exists("/tmp/not_there") == False

General hints

red-green-refactor

A development approach from TDD.

  1. Red - Write a test, it fails
    • Forces us to think about what we want to develop and how do we want to use the interface we’re about to implement.
  2. Green - Make it pass (as fast as possible)
    • If it’s simple, just type it
    • If it’s harder, make a note and write the quickest solution that makes the test pass
  3. Refactor - Spend time to make the implementation correct.

F.I.R.S.T Principles

Tests should be:

  • Fast (encourages us to run them frequently, which increases confidence in the code)
  • Independent (not influence each other)
  • Repeatable (in any environment)
  • Self-validating (a failing test should give enough info about what’s wrong1)
  • Timely written (just before the production code)2

Arrange-Act-Assert (3A)

3A is a common pattern for structuring tests.

  • Arrange -> Create objects / prepare the environment
  • Act -> Simulate behaviour
  • Assert -> Check the results

In a test this would look like this:

string = "ABc"

result = string.upper()

assert result == "ABC"

  1. if you need to look into logs, you should’ve written more tests ↩︎

  2. Not earlier, you need to have context ↩︎

Convert nested OrderedDicts into dict

From SO1, if both are JSON serializable objects, you can use json:

from json import loads, dumps
from collections import OrderedDict

def to_dict(input_ordered_dict):
    return loads(dumps(input_ordered_dict))

  1. python - how to convert a nested OrderedDict to dict? - Stack Overflow ↩︎

Getting screenshots to work in qtile

Get screenshotting working through a hotkey. I need to screenshot an area of the screen, put the screenshot in a folder, and immediately open it.

In i3 had

bindsym Mod3+s --release exec scrot -s -e 'mv $f ~/s/screenshots && eog ~/s/screenshots/$f'

Nothing I tried worked (didn’t do anything weird):

Key([mod], "s", lazy.spawn(CONFIG_LOCATION + "screenshot.sh"))

Tracked it down to two main issues:

  1. scrot works, scrot -s doesn’t. (Running the shell script directly from shell was fine!)
  2. qtile doesn’t like variables in shell scripts
    # this works
    scrot -u -e 'thunar $f' "/tmp/shot.png"
    # this doesn't
    scrot -u -e 'thunar $f' "$SCREENSHOT_PATH/shot.png"
    

Decided to leave the first one alone, scrot -u gets the currently selected window, which generally is good enough for me.

The second one - first rewrote the script to get passed the target path as positional variable (surprisingly it worked!), then decided to do it python-only. As a bonus, copies the screenshot url to the clipboard.

# definition
copy_command = 'bash -c "echo {0} | xclip -selection c"'
# ...
def take_screenshot():
	SCREENSHOT_FILENAME = datetime.now().strftime("qtile_%y%m%d-%H%M%S%z")+"-$w$h.png"
	screenshot_path = D.SCREENSHOT_DIR +"/"+ SCREENSHOT_FILENAME
	command = f"scrot -u -e 'thunar $f && {Commands.copy_command.format('$f')}' {screenshot_path}"
	return command

#usage
Key([mod], "s", lazy.spawn(Commands.take_screenshot()))

(qtile-dotfiles/config.py at master · justinesmithies/qtile-dotfiles has escrotum as python module, errored out during install in the qtile venv and segfaulted on first run when installed outside of it.)

qtile scripting through callables; switching to specific layout

Qtile scripting

Scripting Commands — Qtile 0.1.dev50+ga708c8c.d20211209 documentation has a lot more interesting stuff than the ones exposed through “vanilla” config, finally figured out how to use them:

def test(qtile):
    qtile.cmd_to_layout_index(0)

# ...
Key([mod, ctrl], "apostrophe",  lazy.function(test))

It’s in the docu1 but I missed its significance on first read, then saw hints in a github config2.

The qtile object passed as the first argument is exactly the QTile from scripting.

Qtile switching to a layout by id

To parametrize it, you have to let it return a callable function:

def switch_to(ly_id: int):
    def cb(qtile):
        qtile.cmd_to_layout_index(ly_id)
    return cb

# ...
Key([mod, ctrl], "apostrophe",  lazy.function(switch_to(0))), 

More fun with qtile scripting

I don’t see this mentioned in the docu, but the attributes can be found in the source of libqtile.core.manager — Qtile 0.1.dev50+ga708c8c.d20211209 documentation.


  1. Lazy objects — Qtile 0.1.dev50+ga708c8c.d20211209 documentation ↩︎

  2. https://github.com/qtile/qtile-examples/blob/master/roger/config.py#L34 ↩︎

Restarting qtile when you mess up config file

If you mess up config.py and restart qtile and most of your keybindings aren’t working, if you’re lucky you still have a terminal open. From it, you can fix config.py, then restart via qtile shell -> restart().

211209-1354 Python testing basics with poetry and pytest

(From a python-worshop I attended)

Pytest

Basics

Fixtures for boilerplate code

Fixtures are useful bits you don’t want to repeat every time, like connecting to a database etc.

It’s a function, that may or may not take arguments, that might or might not return something.

Tests can request a fixture, and it’s basically done like this:

@pytest.fixture 
def my_fixture():
	return "fix"

def test_with_fixture(my_fixture):
	assert my_fixture == "fix"

# fixtures inside other fixtures
@pytest.fixture 
def next_fixture(my_fixture):
	return my_fixture + "fix"

They are run independently for each test, to ensure that tests are as separated as possible. There are ways to define their scope, but it’s rarely used.

You can also use them to change settings like logging, by adding a fixture that changes etc.

Marks1 are used to select what you run

“By using the pytest.mark helper you can easily set metadata on your test functions1

Defining marks

Default marks
#@pytest.mark.skip(reason="there's a good reason")
@pytest.mark.skipif(pytorch.cuda.is_available(), reason="there's a good reason")
def test_always_ski():
  assert False

That way you don’t have to do anything inside the test and based on python environment.

Custom marks2
# simple marks
@pytest.mark.whatever
def test_whatever():
  pass

# complex marks (and defined beforehand)
cuda = pytest.mark.skipif(True, reason="...")
@cuda
def test_require_cuda():
  assert False

Marks can be combined

@pytest.mark.one
@cuda
def test_whatever():

Selecting marks when running

Assuming @pytest.mark.gpu:

python3 -m "not gpu"
python3 -m "gpu"
Registering marks 3

Recommended, to keep track of them and get stuff like pytest --markers etc. In pyproject.toml:

[tool.pytest.ini_options]
markers = [
  "gpu: marks test which require a gpu"
]

Mocking

Replace some functions, including ones deep inside code. Lives inside the pypy package pytest-mock · PyPI.

You can patch calls, objects, etc.

from pytest_mock import MockerFixture

def test_mock(mocker: MockerFixture) -> None:
	env_mock = mocker.patch("os.environ.get")
	os.environ.get("something")
	assert env_mock.call_count == 1
# Do stuff to dictionaries:
mocker.patch.dict("os.environ", {"sth": "test"})
assert os.environ.get("sth") == "test"
assert os.environ.get("not_there") == None
# classes, function calls, etc

TODO - does this work for class instances created after the mock?

Spying to keep track of function calls etc

mocker.spy Sample from documentation:

def test_spy_method(mocker):
    class Foo(object):
        def bar(self, v):
            return v * 2

    foo = Foo()
    spy = mocker.spy(foo, 'bar')
    assert foo.bar(21) == 42

    spy.assert_called_once_with(21)
    assert spy.spy_return == 42

Running stuff

Selecting tests 4

  • By filesystem: pytest test_mod.py and pytest testing/
  • By markers: pytest -m mark, pytest -m "not mark"
  • Keywords:
    • pytest -k "MyClass and not method would run TestMyClass.test_something but not TestMyClass.test_method_something
  • Node ids: pytest test_mod.py::test_func or pytest test_mod.py::TestClass::test_method

Useful bits

Loop on fail

pytest-xdist package allows to do pytest --loop-on-fail, which keeps looping tests and you can see the test results in real time

Logging and output

Setting loglevel globally

logger.warning("test") inside tests doesn’t get shown by default, but you can enable this in pytest results:

[tool.pytest.ini_options]
log_cli = true
log_cli_level = "DEBUG"
Setting it for a single test

You can change it in single tests: caplog.set_level(logging.DEBUG)

This is useful if you’re fixing a specific bug and want more logging on a specific test.


  1. Marking test functions with attributes — pytest documentation ↩︎

  2. Working with custom markers — pytest documentation ↩︎

  3. Working with custom markers — pytest documentation ↩︎

  4. Usage and Invocations — pytest documentation ↩︎

Adding a new WM to startup with GDM

To add an item for the WM to the options shown on gdm startup:

  1. Add its .desktop file to /usr/share/xsessions:
[Desktop Entry]
Name=qtile
Comment=Qtile
Exec=/home/me/.dotfiles/qtile/.config/qtile/startup.sh
Type=Application
X-LightDM-DesktopName=qtile
DesktopNames=qtile
Keywords=tiling;wm;windowmanager;window;manager;
  1. sudo systemctl restart gdm.service1

  1. Before that I tried killing gdm3 and X but it didn’t work. ↩︎

211208-1509 qtile WM first impressions

Qtile WM

Python tiling window manager, playing with it for a couple of days now.

It’s everything I expected from a tiling WM, except it’s completely configurable with Python, so basically unlimited options to do anything. Compared to my usual i3: speed is the same, documentation is a bit worse, but configuration is much more intuitive.

And it has a lot of stuff, I never heard of it but was surprised to learn it has a lot of widgets / layouts / etc., and it has  even a CLI-like shell qtile shell where you can use the standard bash commands to do stuff to anything (cd/ls/etc to layouts/groups/windows, run things like cd groups/F1/windows/213; down_opacity()).

Everything I customized in i3 via hacks can be done natively nicely and in python and I love it.

Notes

Checking configuration for errors before restarting

No easy way to check config for correctness I’ve found, but python3 config.py finds most errors.

Docu suggests python3 -m py_compile config.py but it returns no output regardless of errors. qtile shell’s test config also is quiet.

Layouts

A lot of them. Tried all. Favourites so far. Listed here: Built-in Layouts — Qtile 0.1.dev50+g9c583ed.d20211208 documentation

Main realization so far is that I’ve been using tiling WMs wrong, in i3 I kept manually splitting the window when I needed to have it split into smaller ones. Except that this should happen automatically, because I never want three windows side-by-side at the same time.

MonadTall / MonadWide

Probably my favourite one. Splits stuff nicely in one big and multiple smaller ones in a separate columns.

Added these bits to config:

Key([modkey], "i", lazy.layout.grow()),
Key([modkey], "m", lazy.layout.shrink()),
Key([modkey], "n", lazy.layout.normalize()),
Key([modkey], "o", lazy.layout.maximize()),
  • <mod+o> toggles between how big/main is the highlighted window. If it’s the big window, it gets narrower or wider, if it’s one of the smaller ones in a column, each becomes the biggest/smallest in that column.
  • <mod+i>/<mod+m> grows/shrinks the current window.
  • <mod+n> ‘normalizes’ everything by resetting the layout.

Column

Nice intuitive etc, has N columns, moving windows to left-right does what I expect, including creating newer columns, or splitting existing ones as the window “travels” through it.

Bsp

The tree-thingy that splits each thing into two, ad infinitum.

These bindings use mod3 which is the physical ctrl key, that move the splits with all windows inside them (not individual windows). They seem to be used only for that layout.

    Key([mod3], "j", lazy.layout.flip_down()),
    Key([mod3], "k", lazy.layout.flip_up()),
    Key([mod3], "h", lazy.layout.flip_left()),
    Key([mod3], "l", lazy.layout.flip_right()),

Other

Tile

Two stacks, one with N “main” windows (1, but configurable), and a second stack for all the other ones. See no added value compared to the Monad ones. But add_after_last=True makes the behaviour more intuitive to me.

Max

One single window, the rest are hidden behind it (as a stack), no configs, no way to signal if it’s the only window or there are more behind it.

TreeTab

Only layout that I can get to show the titles of the windows inside the stack. You get one stack and window titles on the right.

Meant for browsers like uzbl, and it emulates almost exactly the setup I have for qutebrowser.

Random

  • From this1 sample command:
    • Doing stuff based on different layout:
          layout = qtile.current_layout
      	group = qtile.current_group
      
      	if layout.name == 'monadtall':
      		layout.cmd_maximize()
      		if len(group.windows) != 2:
      			return
      
    • Using python and sound software engineering like creating a class to keep constants for commands

Config bits / settings

Getting Obsidian to run in a Dropdown/Scratchpad

One of those two worked: - calling Obsidian directly as binary (instead of my runner shell script) - Using config.Match()to identify it .

TODO

  • Multiple screens/monitors
    • This shows how to detect number of screens and place groups in them: qtile-examples/groups.py at master · qtile/qtile-examples
      from libqtile.config import Screen
      from platforms import num_screens, hostname
      if num_screens[hostname] == 4:
      	from bars import chat_bar, main_bar, media_bar, code_bar
      	# ...
      	chat_screen = Screen(top=chat_bar)
      	# ...
      	screens = [main_screen, media_screen, code_screen, chat_screen]
      
  • All my usual shortcuts (volume, screenshots, etc. etc.)
  • I like the idea of splitting the configs in separate python files2, especially for constants1

What’s missing

  • How to have a sticky floating window? 3

  1. qtile-config/config.py at eacda219cebe357c46c3708f419f86bb585d4397 · zordsdavini/qtile-config ↩︎

  2. qtile-examples/oboingo at master · qtile/qtile-examples ↩︎

  3. Always on top floating windows · Issue #1260 · qtile/qtile ↩︎

211208-1733 Adding highlights to a list of sample configuration folders

The qtile/qtile-examples: Example configurations and scripts for Qtile have multiple folders with example configurations and in the README a short description of the interesting stuffs there, like “python functions in config” or “themed widgets” or “custom window shortcuts”.

This is really nice on many levels, and helps against the multiple anonymous folders with stuffs one has to click through to find something interesting.

211207-1822 Three ways to return None in python

I can always replace return None with just return in #python. (Third way is omit a return completely.)

More about this: The Python return Statement: Usage and Best Practices – Real Python

211207-2031 Obsidian starring a search

You can star/unstar a search!

Really handy for summary/analysis-type searches, like for hashtags of things that may be reoccurring.

Additionally a “search” doesn’t stop once you click through files or through the folders, it’s still available in its own tab.

Obsidian embedding parts of other document

You can embed not just an entire document, but also part of it, like headers! The same mechanism as with linking, but I can’t figure out how the autocompletion is supposed to be used.

In any case, works the same way, page title and then # for headers and ^ for blocks, for which it will autogenerate a reference in the target file.

To trigger this you have to have the page name already filled in, it suggests stuff, but once you click on something or use tab it generates a link with it immediately. Best way I can figure out is to let it do this, and then replace the syntax around, the autocompletion gets triggered once you get it in a syntax like below: ^66eab0

![[Your page title#

Link

211206-0353 Python multiline regexes

In python, when doing regex on a multiline string:

  • re.MULTILINE makes ^ and $ match on each line, not just begin/end of entire string.
  • re.DOTALL makes . match newline (by default it doesn’t).

Advanced search in Obsidian

Obsidian can do advanced search: Obsidian search

  • “X OR y -z”
  • js-flavour regex
  • Special search operators for files/paths, to search on lines/blocks/sections, tasks, tags

tag: #Tag is better than searching the tag by itself, as the latter might find tags inside code listings etc etc etc

211203-1523 Bitbucket open branch files from PR or commit

When looking at a commit, clicking on “View the entire source for this file” symbol opens that file, and then one can navigate to folders etc as usual, they’ll all be from the current branch.

211203-1941 Obsidian link to headers and internal blocks

Linking to blocks and headers in #obsidian

Is helpfully describeed in the autocomplete for [[:

EDIT 2021-12-07: You can do this from external pages too! Just autocompletion is not intuitive. See 211207-2015 Obsidian embedding parts of other document. 1

Manually creating block references

When linking internally it autogenerates reference names, it seems. ^74ce58

Can I do this? ^myreference

Yes I can! Autocompletion even suggests/uses my reference!

Can _I_ do this? ^myreference

[Yes I can!](#^myreference)  Autocompletion even suggests/uses my reference!


  1. And an internal link to the paragraph: here↩︎

211203-2305 New obsidian Templates + hotkeys for Garden (IT, RL) and personal notes

I changed the templates I use to be more repetitive but hopefully with less chances for a note meant to be private to get published on my website.

Three types of notes I want to be able to create easily:

  • Diensttagebuch (public)
  • Jourrnal (public)
  • Personal (private)

I don’t want the Personal ones to end up left in any of the folders parsed by obyde even by chance, and if they do I don’t want them converted, and if they do - shown.

Now I just create a note, it gets put into /, I give it a name, and then run one of the three templates. The templates take care of moving it to the correct folder and prefic

Now I have three identical templates, they move the note to the correct place, prefix the file with the datetime if needed, and add boilerplate frontmatter.

Public diensttagebuch note (<C-t>), puts it into /garden/it/ and prefixes with datetime:

<% tp.file.move("garden/it/"+tp.date.now("YYMMDD-HHmm")+" "+tp.file.title) %>---
title: "<% tp.file.title %>"
tags:
  - "zc"
  - "zc/it"
  - "<% tp.file.cursor() %>"
fulldate: <% tp.date.now("YYYY-MM-DDTHH:MM:SSZZ") %>
date: <% tp.date.now("YYYY-MM-DD") %>
layout: post
hidden: false
draft: false
---

Public journal note (<C-S-t>) is pretty much identical:

<% tp.file.move("garden/rl/"+tp.date.now("YYMMDD-HHmm")+" "+tp.file.title) %>---
title: "<% tp.file.title %>"
tags:
  - "zc"
  - "zc/rl"
  - "<% tp.file.cursor() %>"
fulldate: <% tp.date.now("YYYY-MM-DDTHH:MM:SSZZ") %>
date: <% tp.date.now("YYYY-MM-DD") %>
layout: post
hidden: false
draft: false
---

Notes not meant to be published (<C-t>) get put into /Personal , but also:

  • Have no date in frontmatter, obyde should loudly error out if it sees them (which it should never)
  • If they magically end up published, I put literally all “don’t pulbish me” parameters in the header.

211202-0008 Hugo and HTML anchors

Hugo generates anchors from headers automatically 1. Tested it - yes, except they’re lowercased and spaces get converted to - (which makes sense).

As a refresher, in HTML it’s

<h2 id="anchor">..</h2>
<a name="anchor"></a>

<a href="#anchor">anchor link </a>

  1. Creating anchors in Hugo pages [SOLVED] - support - HUGO ↩︎

211201-0057 Obsidian daily notes templates

This python script is similar to my approach; the file then is used in obsidian.

I like the idea of adding the weather there! And of getting TODOS by parsing md files. And getting the calendar

Python really is the better thing to use for that, not bash. Sadly needs a computer to run.

Link from https://jamierubin.net/2021/04/01/my-obsidian-daily-notes-automation-script-is-now-available-on-github/

When looking for similar stuff stumbled upon https://forum.obsidian.md/t/slrvbs-journaling-setup/22346, an example of how deep can one go with obsidian templates only.

211201-1637 mypy and python typing

One additional way to check the type hints in #python is mypy, installable as python package.

mypy -p package_name checks the typing in the package, and found some potential errors in corner cases I didn’t know about in one of the projects I’m working on!

Finds wrong typing, missing/wrong return values, that kind of stuff.

It doesn’t like what: str or Path typing output, I guess only Union[str, Path] - is there a reason for it?

In any case I like it more than Pycharm’s way of outputting things and will be using it along with black and flake8 in the future (along with typing itself).

#py/mypy

211201-2236 job interview questions

“What’s the best part and what’s the worst part about working at $companyName for you?”

(Asked by a candidate in a job interview I was present on, during the usual “do you have any questions about our company” bit)

211201-2243 things to learn

No particular order, just things that keep creating problems because I never learned then for real:

  • Git
    • From the ground up through command line
    • Get a better picture of how pycharm maps to this
    • Learn to use tig better
  • Python
    • standard library
    • the ‘new’ python 3.7-3.9 features
  • German

Stuff to learn in my free time

Originally written 2016-06-06 13:14:53+00:00, quoting in full:

Один ответ на Кворе 1 стал вот последним катализатором, все это крутилось в голове уже пару лет, настало время формализировать это, пусть  с “крутилось в голове” это станет конкретным чеклистом. Потом (но потом) — конкретными планами уровня “до конца месяца я учу Х”

  1. Learning how to learn
    1. That Coursera course
    2. A couple of books on the topic
      1. Re-read “The art of learning”
  2. Mental frameworks
    1. Теория практического мышления” — законспектировать
    2. Systems theory
      1. Grok “Thinking in systems – a primer” — тоже законспектировать и запостить сюда
      2. Много думать об этом in real life, искать примеры, наблюдать за миром from the perspective of a systems theorist
    3. See https://www.farnamstreetblog.com/mental-models/ and research them
      1. Make a big post here with the full list and good explanations
    4. From this answer about being an expert-generalist there’s “Poor Charlie’s Almanack: The Wit and Wisdom of Charles T. Munger” (goodreads)
    5. Polya — “How to solve it”
    6. Adler — “How to read a book”
    7. Game theory and rationality
      1. Lesswrong’s Core Sequence
      2. Not to learn formally, but regardless freaking interesting resources for my spare time:
        1. https://www.scotthyoung.com/ !
        2. https://feeds.feedburner.com/brutereason
        3. http://slatestarcodex.com/feed/ !
        4. https://feeds.feedburner.com/MeltingAsphalt
  3. Improve my memory
    1. “The Memory Book: The Classic Guide to Improving Your Memory at Work, at School, and at Play” 2

    2. Remember Everything You Want and Manage the Rest: Improve Your Memory and Learning, Organize Your Brain, and Effectively Manage Your Knowledge 3

  4. Productivity and stuff
    1. I hear good things about “Deep Work: Rules for Focused Success in a Distracted World” (“These books should be taken together as a whole because they give you the WHAT, the WHY and the HOW for being an elite knowledge worker.”)
    2. “Getting Things Done: The Art of Stress-Free Productivity”, a legend
  5. Philosophy, the very basics; Psychology, the very basics.
    1. Leaving aside the “everything worth doing is worth doing well”, my old project of “reading from the very beginning” is still okay; I’ll need time, but I will do it.
    2. BUT the absolute priority in this is to reread and rethink Seneca and Marcus Aurelius

All this on a thread by itself. I will work on my professional skills independently; that means mostly social engineering and infosec. For infosec there’s this, for SE I should at least working in my uni courses.

I mention a lot of books which, traditionally, I would need to buy in paperback to be able to highlight and write on them. Bad on finances, but the 10% rule is still with me; also, objectively/strategically speaking, it’s a very good investment in my future.

Speaking of books: https://www.quora.com/What-are-some-books-that-expand-our-mind/answer/Marcus-Geduld (“What are some books that expand our mind”) — a very thorough answer.


  1. https://www.quora.com/What-is-the-most-useful-skill-I-can-develop-in-3-months ↩︎

  2. https://www.amazon.com/gp/product/B006Q1SQCQ?ref%5F=oh%5Faui%5Fsearch%5Fdetailpage&redirect=true&psc=1&pldnSite=1 ↩︎

  3. https://www.amazon.com/gp/product/B00CL8F7EI?ref%5F=oh%5Faui%5Fsearch%5Fdetailpage&redirect=true&psc=1&pldnSite=1 ↩︎

Tibetan bowl 5min video

Since I keep looking for it and it’s so hard to search for: ❀ Tibetan Bowl - Every 5 Minutes - YouTube

Самые вкусные в мире котлеты

Первая часть

  • Берешь сковородку,

  • немного подсолнечного масла,

  • на нем жаришь 2 головки чеснока (подавленного)

  • к нему добавляешь перевжуханные на блендере 1 большую или 2 маленькие луковицы

  • когда лук пропассируется добавляешь туда тертую 1 большую или 2 маленькие морковки

  • оставляешь остывать

  • Варишь немного риса любым способом найденным в интернете. Оставляешь его остывать

Сами котлеты

  1. Смешиваешь то что на сковородке с 1 стаканом риса с фаршем, солишь и 1 чайная ложка сахара
  2. “Отбиваешь котлеты”
    • Берешь кусочек фарша, отводишь на где-то 20 см и кидаешь обратно в фарш
  3. Жаришь без масла на большой сковородке, пока вся котлета не будет одного цвета
    • ~10 мин каждую сторону

211130-1751 git rebase on branch vs origin-branch + git fetch

Had issues, asked for help, and then learned a lot of stuff.

git rebase branchname != git rebase origin/branchname!

The first one is about the current local state of the branch, the second one about the one on remote.

BUT the one on remote as known by local != one on remote-remote! You need to update first!

git fetch --all or whatever.

I’d previouly update / pull before through PyCharm before doing that, and this abstracted all of this away from me.

211130-1925 providing user and pass to wget through teamcity

Tried to download a Teamcity artifact through wget, and apparently you can if you provide a user/pass through wget!

I assume it’s HTTP auth or something

wget --user username --password my-password https://teamcity.location/repository/download/....

211129-0023 obsidian console

To access the #obsidian console, <C-S-i> worked. It was the standard “Dev tools”.1


  1. How to access the console? - Help - Obsidian Forum ↩︎

Obsidian starter templates

This contains #obsidian starter templates for a couple of scenarios (link database, research, ….)

This amount of stuff available for Obsidian really gives me hope it will live on.

masonlr/obsidian-starter-templates: Starter templates for Obsidian.md

211128-2120 simple-scan for scanning

Since I seem to keep forgetting: simple-scan is the program I use to talk to scanners. You can select various options (scan document, photo etc).

Keeps #scanning in the exact same PDF document until you break it.

211126-1301 pycharm pinning tabs

In #pycharm, “Pin tab” exists! But then it’s not “Tab 1” etc anymore and I can’t use my shortcuts

LOT Polish Airlines

Had working outlets in the last 4 planes UA <-> PL <-> DE! 220v + USB #triggers/trip/airports

211124-1731 python logging setup

From a conversation with a colleague at work about #py/logging

Naming loggers after the package / files

Logger names can be used to cleanly output and separate them.

Assuming one has a package with multiple files/subfolders in it, it’s possible to give each one their own logger, like this:

In the main file of the package:

logger = logging.getLogger(__package__)

In all the other files:

logger = logging.getLogger(__name__)

That way paths ./package/my_module.py lead to loggers named like package.my_module that map the semantical and the directory structure.

Changing settings of the loggers

In a setup above, one can then easily change the settings of the loggers referring to them by their names.

Configuring logging: Logging HOWTO — Python 3.10.0 documentation

Changing loglevel is easy from code,

if args.debug:
		logger.setLevel(logging.DEBUG)

logging.config allows to change the config from ini-like config files. Two main ways: logging.config.fileConfig reads ini-like config files, logging.config.dictConfig 1 from dictionaries.

Sample .yaml that when converted to dict would change the loglevel of different loggers:

version: 1
                               
loggers:
	packageName.mymodule1:
		level: DEBUG
	packageName.mymodule2:
		level: DEBUG

These loggers can even include external ones!


  1. logging.config — Logging configuration — Python 3.10.0 documentation ↩︎

211124-1744 argparse notes

(Those too after a long talk to a colleague at work, this time #py/argparse)

Cool things about argparse:1

  • parser.add_argument('--two-words') would automatically map to args.two_words (_ vs -)!
  • One can provide complex types!2 For files, two options.
    • The first one allows to set file permissions etc., but it opens them and returns the handle to you, which you may not want.
    • pathlib.Path() works as expected, and even automagically parses string paths from args into the Path!
      • Additionally we can then establish that we’re working with Paths from the very beginning, getting rid of the str or Path ambiguity.
      • “Be strict and clear from the very beginning, then you don’t have to deal Path or str”

    • Sample of both from official documentation:
      parser.add_argument('a', type=argparse.FileType('w', encoding='latin-1'))
      parser.add_argument('b', type=pathlib.Path)
      
  • You can get defalut values from os.environ()! Then you can also run it as
    WHATVEER_VALUE=234 python3 file.py
    

A nice structure for it all is:

  1. if __name__ == '__main__': runs a function like main() getting rid of the scope issues
  2. Parsing is done my a separate function, that returns the Namespace:
    def parse_args() -> argparse.Namespace:
        parser = argparse.ArgumentParser()
        parser.add_argument('--input-directory' ..)
        return parser.parse_args()
    
  3. Then in main() we use it like args = parse_args(); if args.input_directory == ... This is nice also because then we don’t have to deal with an arparse object in main, just its results.

Also, in general, CLI programs have arguments like program --arg-one, not program --arg_one. I write the latter one because I still feel I’m in a python world, but Python would parse such dashed arguments into classic ones (see above). TODO look for some best practices for CLI programs, including Python ones, POSIX etc etc etc.


  1. argparse — Parser for command-line options, arguments and sub-commands — Python 3.10.0 documentation ↩︎

  2. argparse — Parser for command-line options, arguments and sub-commands — Python 3.10.0 documentation ↩︎

211123-2122 obsidian undeleting files

If sync is enabled, in settings -> Sync there’s a “Deleted files” with versions and actions.

If not, unless a setting is set to delete to Obsidian’s trash, it’s left to the filesystem, so trash can or extundelete in my case or whatever.

211123-2333 python packaging and poetry

(From MO’s python riddle at work)

Things declared in if __name__ == '__main__' are in global scope. Not because it’s special, but because ..global scope.

Code from SO answer:[^2]

In main:

>>> if __name__ == '__main__':
...     x = 1
... print 'x' in globals()
True

Inside a function:

>>> def foo():
...     if __name__ == '__main__':
...         bar = 1
... foo()
... print 'bar' in globals()
False

Python doesn’t have block-local scope, so any variables you use inside an if block will be added to the closest enclosing “real” scope.

Someone mentioned that if __name__ == '__main__' can happen anywhere in the code. Never thought about this

211123-2333 python scopes

(From a python riddle at work)

Things declared in if __name__ == '__main__' are in global scope. Not because it’s special, but because ..global scope. All these bugs go away if you move main() to a separate function.

Code from SO answer:[^2]

In main:

>>> if __name__ == '__main__':
...     x = 1
... print 'x' in globals()
True

Inside a function:

>>> def foo():
...     if __name__ == '__main__':
...         bar = 1
... foo()
... print 'bar' in globals()
False

Python doesn’t have block-local scope, so any variables you use inside an if block will be added to the closest enclosing “real” scope.

Someone mentioned that if __name__ == '__main__' can happen anywhere in the code. Never thought about this

211123-2345 python packaging

Providing a __main__.py along with __init__.py makes the package itself executable:

$ python -m module_name

__main__.py would have an usual if __name__ == "__main__" block and run stuff imported from other files of that package.

211123-2348 poetry for package management

Short notes about #py/poetry for package management

poetry new packagename creates a poetry project

From within the folder with the package:

  • poetry install == pip3 install -r requierements.txt
  • poetry shell == source .venv/bin/activate
  • exit == deactivate

Basic usage | Documentation | Poetry - Python dependency management and packaging made easy:

  • venvs live in {cache-dir}/virtualenvs, which on my box is /home/me/.cache/pypoetry/virtualenvs/ptest-eeSDLvcF-py3.6/bin/activate
  • poetry.lock caches the resolved packages once we install things once.
    • Must mach pyproject.toml, a warning will be shown otherwise
    • It’s important to commit it to a VCS! It has the exact versions it resolves, beneficial for everyone to use them
  • poetry update updates everything to the latest versions, overwriting poetry.lock
  • poetry init initializes a project and creates a pyproject.toml interactively, allowing even to search for packages etc!

Adding packages:

  • poetry add yaml adds a package
  • poetry search yaml looks for packages in remote repos! Will tell you that you actually want pyyaml

211122-0256 quickly forming an URI in markdown

Found this in old markdown code from my old blog, I guess I forgot about this:

<what@ever.com>
<https://example.com>

211122-0905 detectron Instances initialization

Detectron’s Instances object gets created like this, creating attributes with names unknown initially:

def __init__(self, image_size: Tuple[int, int], **kwargs: Any):
    """
    Args:
        image_size (height, width): the spatial size of the image.
        kwargs: fields to add to this `Instances`.
    """
    self._image_size = image_size
    self._fields: Dict[str, Any] = {}
    for k, v in kwargs.items():
        self.set(k, v)

Which is neat.

To create an Instances object for unit tests I did:

pred_boxes = Boxes(tensor(
[
    [ 143.8892, 1166.6632, 1358.7292, 1411.6588],
    [ 131.3727,  864.3126, 1355.7804, 1144.3668],
    [ 585.6373,  747.7184,  922.6433,  815.9998]
]))
scores = tensor(
    [0.9971, 0.9967, 0.9938]
)
pred_classes = tensor([3, 3, 3])

instances = Instances(
    image_size=(2122, 1500),
    scores=scores,
    pred_classes=pred_classes,
    pred_boxes=pred_boxes
)

211121-2123 git undoing git add unstaging files


title: “211121-2123 Undoing git add / unstaging files” tags:

  • “zc”
  • “zc/it”
  • “git” fulldate: 2021-11-21T21:11:47+0100 date: 2021-11-21 layout: post hidden: false draft: false

Two different questions here! Both options are: 1 If you add a file for the first time, git rm --cached . or git -rm -r --cached . will reverse that.

If you want to un-add changes to a file that’s already in the repo, git reset <file> / git reset will undo that.


  1. How do I undo ‘git add’ before commit? - Stack Overflow ↩︎

211121-2201 vim opening more than 10 tabs

When opening a lot of files as vim -p *.md* only 10 kept being opened, finally googled it.

Solution: adding set tabpagemax=50 to ~/.vimrc

Old blog - randomness of this week

date: 2013-10-03T16:52:37+00:00
guid: http://shmt.pp.ua/blog/?p=326

Nowplaying: Amber Route – “Don’t drink lemonade formaldehyde”

Музика, про яку не варто було б забути:

  • Amber route
  • September Malevolence
  • Franco Battiato
  • Аквариум
  • Smokie

Нові слова:

  • Ерзац (ersatz) — німецьке слово, що буквально означає заміну (замінюючи) або заміни[1]. Продукт, зазвичай, поступається за якістю, наприклад, «цикорієва ерзац-кава»; неповноцінний замінник.

    • (вперше зіштовхнувся з ним у Чубая ще влітку, не мав під рукою словника)
  • In the United States, a furlough (/ˈfɜrl/; from Dutch: “verlof”) is a temporary unpaid leave of some employees due to special needs of a company, which may be due to economic conditions at the specific employer or in the economy as a whole

  • Thanatology is the scientific study of death.

Programming is

From old blog, date: 2015-11-29T13:49:32+00:00

  • Programming as a way to do brilliant things.
  • Programming as an art.
  • Programming as a way to challenge my skills.
  • Programming as a way to change the world.
  • Programming as a way to be free.
  • Programming as a way to have a purpose.
  • Programming as a way not to be afraid — “Я могу все.”
  • Programming as a way to improve myself.
  • Programming to BE.
  • Programming as an awesome thing to know.
  • Programming as magic.
  • Programming as a way to solve many problems.
  • Programming as beauty.
  • Programming as learning; programming as hunger for knowledge.
  • Programming as playing God.
  • Programming as constant self-improvement.
  • Programming as mindset.
  • Programming as opportunity to understand how complex systems work.
  • Programming as opportunity to understand how complex systems can be made work differently than intended.
  • Programming as a medium of expression.

Quotes from my old blog

From my old blog:

Я никогда не могла понять, почему так мало людей не могут мне ответить на вопрос: “Чего ты хочешь?” Это серьезно какой-то баг в большинстве людей. Свои цели на будущее нужно осознавать предельно четко, и периодически вспоминать о них. (с) М.

“Keeping options open” often means “not being successful in anything” ((https://www.quora.com/How-do-you-decide-what-to-do-when-you-are-multitalented-and-want-to-have-all-of-it/answer/Shulamit-Widawsky))


W/o sources:

Whenever Jobs suffered a setback, it wasn’t too long before he was looking forward and thinking about what he should do next.

“What’s the point in looking back,” he told me in one email. “I’d rather look forward to all the good things to come.”- Brent Schlender and Rick Tetzeli, Becoming Steve Jobs

Celebrate other people’s success and you’ve accomplished, but raise the bar a little higher each time you succeed.

Refuse to accept that something is beyond your understanding. Sometimes, you have to learn a set of special codes and symbols to get at the knowledge you want. You cannot allow language acquisition, whether of spoken  communication or a set of mathematical, electric, or other scientific code, to daunt you.

First find a good direction, and once you’re sure you’re going the right direction, start going faster. First be resultative, then increase productivity; Success is measured in output of value, not input of effort. + http://www.arhivstatey.ru/articles/organizer/index.html

Two kinds of self-confidence

From old blog:

date: 2016-09-19T15:36:09+00:00

There are two forms of confidence: contextual — how well you can do something particular (driving, public speaking, …), and general — the confidence that no matter in which unknown situation you are thrown in, you’ll find a way. The second kind is necessary to continually try new things. (from http://highability.org/756/using-your-multipotentiality-to-grow-your-confidence/)

Fail fast and iterate

211120-2101 Шорох орехов

From Атомный век : Забавные истории — История Росатома:

Шорох орехов

Курчатов дал задание группе молодых ученых — составить план разработки термоядерной реакции. После нескольких дней жарких обсуждений и споров план был готов, и молодежь принесла его академику.

— Шорох орехов, — сказал Игорь Васильевич, пробежав глазами план. — Не знаете, что такое шорох орехов? Сидит грузин, торгует орехами за ту же цену, за которую их купил. «Какой смысл так торговать?» — спрашивают его. «Люблю шорох орехов!». Вот и у вас получилось то же самое.

И отправил план на доработку.

Не наш груз

В кабинете директора ядерного центра ВНИИТФ генерала Ломинского однажды раздался телефонный звонок.

— Георгий Павлович, на железнодорожной станции в Челябинске произошел мощный взрыв. Не ваш ли груз взорвался в вагоне?

— Челябинск цел?

— Цел.

— Значит, не наш.

211119-1208 BER airport charging stations

BER (#Berlin Brandenburg) airport has a lot of charging stations in gate area B. And none in the central/food area.

211118-0024 python namedtuple

Python’s NamedTuple is really cool!

Python’s Instance, Class, and Static Methods Demystified – Real Python is an excellent guide, as is the entire website.

NamedTuple VS Dataclass, copying from SO answer:[^1] When your data structure needs to/can be immutable, hashable, iterable, unpackable, comparable then you can use NamedTuple. If you need something more complicated, for example, a possibility of inheritance for your data structure then use Dataclass.

The immutable part is important - can’t do named_tuple.value = 3 after creating it.

Can be created also through colections.namedtuple, copied directly from :

>>> from collections import namedtuple

>>> Person = namedtuple("Person", "name children")
>>> john = Person("John Doe", ["Timmy", "Jimmy"])
>>> john
Person(name='John Doe', children=['Timmy', 'Jimmy'])
>>> id(john.children)
139695902374144

211118-1832 mob programming and mob review

(heard at work)

The basic concept of mob programming is simple: the entire team works as a team together on one task at the time. That is: one team – one (active) keyboard – one screen (projector of course).

— Marcus Hammarberg, Mob programming – Full Team, Full Throttle1

“"Mob programming is a software development approach where the whole team works on the same thing, at the same time, in the same space, and at the same computer. “Mob code review is a software development approach where the whole team reviews on the same thing, at the same time, in the same space, and at the same computer.”2


  1. Mob programming - Wikipedia ↩︎

  2. From no code review to mob code review | by Nicolas Dupont | Akeneo Labs | Medium↩︎

211117-1127 python simple TTL time-based caching

functools has lru_cache, really easy to add it as decorator to a function to cache the responses! Example directly copied from caching - Python in-memory cache with time to live - Stack Overflow:

from functools import lru_cache
import time


@lru_cache()
def my_expensive_function(a, b, ttl_hash=None):
    del ttl_hash  # to emphasize we don't use it and to shut pylint up
    return a + b  # horrible CPU load...


def get_ttl_hash(seconds=3600):
    """Return the same value withing `seconds` time period"""
    return round(time.time() / seconds)


# somewhere in your code...
res = my_expensive_function(2, 2, ttl_hash=get_ttl_hash())
# cache will be updated once in an hour

Used it practically in some code that called an expensive external function multiple times. Bad code I didn’t have time to fix, but it took 2.5 seconds to run. Adding the lines above shortened the runtime from ~2.5 seconds to 0.02 seconds with cache lifetime of 60 seconds.

Didn’t update the function at all without the del ttl_hash and default none parameter bit, TODO understand what’s really happening there.

211117-1251 etcher is a program to burn ISOs on usb drives

balenaEtcher - Flash OS images to SD cards & USB drives is mentioned in the official Mint installation guide1 and is quite neat!

No support for persistant storage like the good old unetbootin, but I guess still higher-level than dd.


  1. Create the bootable media — Linux Mint Installation Guide documentation ↩︎

211117-1304 delete all empty files in folder

find -size 0 -print -delete , or find /foldername -size 0 -print -delete .1


  1. filesystems - Linux delete file with size 0 - Stack Overflow ↩︎

211117-1309 obsidian plugin footnote shortcut

Added “Obsidian footnotes1” plugin, bound it to <C-R>, adds numbered footnotes. Emulates my old vim footnote macro, except that footnotes are numbered and therefore automatic.

Ideally (for the master page, hypotetical merging of markdown files) I’d allow for non-automatic ones as I had in vim (I type whatever, press the footnote shorcut, creates a footnote with index whatever) and this would be a nice case for a simple obsidian template but I won’t be doing it in the near term.


  1. akaalias/obsidian-footnotes: Makes creating footnotes in Obsidian more fun! ↩︎

211117-1415 Pycharm / intellij reopen closed tab + current keymap

Pycharm / intellij idea have an action called “Reopen closed tab”. Set it to <C-S-T> a la Chrome, works nicely!

There’s also a default <C-A-left> shortcut for last cursor location1 that does the same.

My current keymap looks like this:

Short Summary
<keymap version="1" name="XWin copy" parent="Default for XWin">
  <action id="ActivateCommitToolWindow">
    <keyboard-shortcut first-keystroke="shift alt 3" />
  </action>
  <action id="ActivateDebugToolWindow">
    <keyboard-shortcut first-keystroke="shift alt 2" />
  </action>
  <action id="ActivateFavoritesToolWindow" />
  <action id="ActivateFindToolWindow" />
  <action id="ActivateMessagesToolWindow" />
  <action id="ActivateProblemsViewToolWindow">
    <keyboard-shortcut first-keystroke="shift alt 4" />
  </action>
  <action id="ActivateProjectToolWindow">
    <keyboard-shortcut first-keystroke="shift alt 1" />
  </action>
  <action id="ActivateRunToolWindow" />
  <action id="ActivateServicesToolWindow" />
  <action id="ActivateStructureToolWindow" />
  <action id="ActivateTODOToolWindow">
    <keyboard-shortcut first-keystroke="shift alt 5" />
  </action>
  <action id="ActivateVersionControlToolWindow" />
  <action id="CheckinProject">
    <keyboard-shortcut first-keystroke="ctrl k" />
    <keyboard-shortcut first-keystroke="ctrl alt c" />
  </action>
  <action id="DuplicatesForm.SendToLeft" />
  <action id="DuplicatesForm.SendToRight" />
  <action id="EditorDown">
    <keyboard-shortcut first-keystroke="down" />
    <keyboard-shortcut first-keystroke="altGraph t" />
  </action>
  <action id="FileChooser.GotoHome" />
  <action id="FileChooser.GotoModule" />
  <action id="FileChooser.GotoProject" />
  <action id="FindNext">
    <keyboard-shortcut first-keystroke="f3" />
  </action>
  <action id="GotoTest" />
  <action id="IntroduceConstant" />
  <action id="MoveEditorToOppositeTabGroup">
    <keyboard-shortcut first-keystroke="ctrl alt l" />
  </action>
  <action id="NextSplitter">
    <keyboard-shortcut first-keystroke="ctrl l" />
  </action>
  <action id="PrevSplitter">
    <keyboard-shortcut first-keystroke="ctrl h" />
  </action>
  <action id="ReformatCode" />
  <action id="ReopenClosedTab">
    <keyboard-shortcut first-keystroke="shift ctrl t" />
  </action>
  <action id="ServiceView.ShowServices" />
  <action id="Switch To Last Tab">
    <keyboard-shortcut first-keystroke="alt period" />
    <keyboard-shortcut first-keystroke="alt 0" />
  </action>
  <action id="Switch To Tab #1">
    <keyboard-shortcut first-keystroke="alt 1" />
  </action>
  <action id="Switch To Tab #10">
    <keyboard-shortcut first-keystroke="alt 0" />
  </action>
  <action id="Switch To Tab #2">
    <keyboard-shortcut first-keystroke="alt 2" />
  </action>
  <action id="Switch To Tab #3">
    <keyboard-shortcut first-keystroke="alt 3" />
  </action>
  <action id="Switch To Tab #4">
    <keyboard-shortcut first-keystroke="alt 4" />
  </action>
  <action id="Switch To Tab #5">
    <keyboard-shortcut first-keystroke="alt 5" />
  </action>
  <action id="Switch To Tab #6">
    <keyboard-shortcut first-keystroke="alt 6" />
  </action>
  <action id="Switch To Tab #7">
    <keyboard-shortcut first-keystroke="alt 7" />
  </action>
  <action id="Switch To Tab #8">
    <keyboard-shortcut first-keystroke="alt 8" />
  </action>
  <action id="Switch To Tab #9">
    <keyboard-shortcut first-keystroke="alt 9" />
  </action>
  <action id="TodoViewGroupByFlattenPackage" />
  <action id="TypeHierarchy" />
  <action id="TypeHierarchyBase.BaseOnThisType" />
  <action id="Vcs.Log.FocusTextFilter" />
  <action id="Vcs.ReformatCommitMessage" />
  <action id="com.mikejhill.intellij.movetab.actions.MoveTabLeft">
    <keyboard-shortcut first-keystroke="shift ctrl page_up" />
    <keyboard-shortcut first-keystroke="ctrl comma" />
  </action>
</keymap>

  1. How to reopen the latest closed files – IDEs Support (IntelliJ Platform) | JetBrains ↩︎

211117-1803 pycharm debugging scrolling

The running tests window has options, like “select first failed test on completion” and “scroll to end”.

211117-1926 python staticmethods and self

I should make use more often of the fact that @staticmethod and @classmethod methods can be called as self.mystaticorclassmethod() in the “standard” methods.

(Another installment of “I should use tree more”)

211117-2107 added sort by size alias

Added this to ~/.zshrc, since I seem to type it so often to have memorized it:

alias dus="du -hd1 | sort -h"

Returns the sizes of dirs sorted by size:

32K	    ./configs
5,2M	./small_dataset
24M	    ./conversion
630M	./model
792M	.

211117-2112 df for current filesystem or speficied file

TIL df -h filename (or more likely df -h .) returns the info about the filesystem that file is in. Will save me a lot of time, since usually that’s exactly teh one I need.

Story behind this: Mistyped df -h as df -, it returned:

Filesystem                  1K-blocks      Used Available Use% Mounted on
/dev/mapper/ubuntu--vg-root 488960032 463006852   1045612 100% /

Wanted to find out what happened. Likely this:

  • - in zsh is the last directory you were in (since cd - does gets you there)
  • man df says that:
     df displays the amount of disk space
           available on the file system containing each file name argument.  If no file name is given,
           the space available on all currently mounted file systems is shown.
    
  • -> It was showing the file system the previous dir was in, which was the current filesystem.

211117-2327 python annotating number of elements in Tuple, Sequence, List in typing

Based on two SO answers1 2:

  • whatever: List[str,str,str] can’t be done, because lists inherently change size
  • if you know the size beforehand, use a tuple, that can be parametrized like that
  • In general, named tuples 3 are really cool in such scenarios

  1. python - How to define the size of a tuple or a list in the type hints - Stack Overflow ↩︎

  2. type hinting - Specify length of Sequence or List with Python typing module - Stack Overflow ↩︎

  3. Write Pythonic and Clean Code With namedtuple – Real Python ↩︎

211110-1520 Historical document processing, dhSegment

This is really cool and of course historical document processing is an established research area: Introduction — dhSegment documentation

211109-1539 Git tracks executable bit of files

Git doesn’t track permissions, except whether the file is executable for the current user. 1

To recursively set all files (but not directories, because then you can’t ls them…) to not-executable:

find . -type f -print0 | xargs -0 chmod -x

To unset this for current repo (--global to unset this globally):

git config --local core.fileMode false

  1. How Git Treats Changes in File Permissions. | by Tah Teche | Medium ↩︎

211108-1203 RabbitMQ

RabbitMQ is a message broker / scheduler that allows sending/receiving messages.

RabbitMQ is a message broker: it accepts and forwards messages. You can think about it as a post office: when you put the mail that you want posting in a post box, you can be sure that the letter carrier will eventually deliver the mail to your recipient. In this analogy, RabbitMQ is a post box, a post office, and a letter carrier.

The major difference between RabbitMQ and the post office is that it doesn’t deal with paper, instead it accepts, stores, and forwards binary blobs of data ‒ messages.

211108-1212 nvidia-smi has a python library (bindings)

nvidia-smi has a python library: nvsmi · PyPI

import nvsmi

nvsmi.get_gpus()
nvsmi.get_available_gpus()
nvsmi.get_gpu_processes()

211108-1246 Hugo groupBy to group stuff by days

Previously I had the posts split by days (“Day 1234”), now for every former h2-header I have a separate post, but still want to split them by days.

Hugo can group posts by stuff, including by dates. 1

This kinda works with pagination. 2

Now my list.html template for Diensttagebuch uses this to iterate through days/groups:

{{ $pages_k := where .RegularPagesRecursive ".Parent.Title" "Days" }} 
{{ $pages_j := where $pages_k "Params.draft" "ne" true}} 
{{ $pages_l := where $pages_j "Params.hidden" "ne" true}} 
{{ range (.Paginate ($pages_l.GroupByDate "2006-01-02")).PageGroups  }}

With the important bit being here, this iterates by day, not by month as in the examples: $pages_l.GroupByDate "2006-01-02"

Then the “day” header itself is {{.Key}}, to get the day of the month + month-year I do this:

<span class="day">{{ dateFormat "02" .Key }}</span>
{{ dateFormat "Jan 2006" .Key }}

Then iterating through the individual posts inside each “day” is:

{{ range .Pages }}
    <a href="{{ .RelPermalink }}">{{.Title}}</a>
    <span class="description">
    {{ .Content }}
    </span>
{{ end }}

  1. Everything that has to do with grouping and lists described here: Lists of Content in Hugo | Hugo↩︎

  2. Pagination | Hugo ↩︎

211108-1316 Syntax highlight of Hugo templates in code listings

“Hugo uses Go’s html/template and text/template libraries as the basis for the templating.” 1

I tried to use go as “language” in code blocks to highlight Hugo templates and it seems to work nicely!

The result of

```go
{{ range (.Paginate ($pages_l.GroupByDate "2006-01-02")).PageGroups  }}
```

is

{{ range (.Paginate ($pages_l.GroupByDate "2006-01-02")).PageGroups  }}

(I generated the first code listing using the \{-{-< highlight go >\}\} Hugo shortcode)


  1. Introduction to Hugo Templating | Hugo ↩︎

211108-1405 Hugo create shortcode or template for Day

Goal: convert “2010-01-01” into “Day 1234”.

First tried to create a Hugo shortode, but you can’t use a shortcode inside a template:

Process: loading templates: ".../index.html:84:1": parse failed: template: index.html:84: unexpected "<" in command

Next step - a partial template! To call them one uses {{ partial templatename .}}, with . being the “context”. I passed .Key, that has the groupBy date, and it works.

So, the partial template day.html does ugly math to get the number of days since the first day of 2019:

{{ $date := (printf . | time) }}
{{ $startUnix := (printf "2019-01-01" | time) }}
{{ $diff := sub $date.Unix $startUnix.Unix }}
{{ $diffInDays := div $diff 86400}}
{{ $diffInDays }}

Then I use it inside templates like this:

<h2 class="title day">
{{ partial "day.html" .Key }}
</h2>

Chicken brining

Brining means putting it in saltwater for some time. Makes everything better.

Really nice simple recipe that includes brining: Baked Chicken Breast | Gimme Some Oven

  • 230C, 15-18 min

  1. How to Salt Chicken & How to Brine Chicken ↩︎

English phrases about time estimates

  • “Any feeling about how much work is left for xx?” - RF
  • “We see a lot of chances for simplification and refactoring” - MO

Medicine and IT and approaches to mistakes

To have open communication while making few errors, have a culture where you can ask questions without judgement before doing stuff.

For example, it should be okay to ask “can I give Ibuprofen to pt. Y with diagnosis Z?” without thinking stuff like “will someone think I don’t know Ibuprofen, diagnosis Z or medicine?”

(from VG, who in turn took it from a lecture about Fehlerkultur in a semi-medical context)

Pergamon, keychains and Pavlovian reflexes

When visiting the Museum of Islamic Art in Berlin, I was playing with my (metallic) keychain. A worker there found me (“AHAAA, it was YOU!") and told me this charming story:

Their boss has a similar metallic keychain, he also loves to play with it, and that the chain’s sound makes most museum workers extremely worried. And that I kept the guy tense throughout most of the eight and ninth century.

When to hold on to information + problems and solutions

AR: “When I tell people about a problem, I always try to offer possible solutions” (In the context of when telling people about a problem)

This reminds me of this snipped from my notes, sadly I don’t have the source anymore:

Managing information 1Imagine that on Monday afternoon your doctor gets the bloodwork results from your recent checkup and sees possible early signs of cancer. Most people would want to know about the doctor’s assessment as soon as possible. They might even say the doctor has an ethical responsibility to share that information. But what if the doctor knows she’ll be getting a more detailed results the next day that can confirm or deny her concerns? Add in that caveat, and now it seems that maybe a doctor shouldn’t scare the hell out of her patients until she has the facts she needs – especially if it only means waiting another day or two. But does that preference change yet again if the results won’t be ready for two more weeks?

When to Hold onto Information

If the following conditions are met, it may be better to hold off explaining a new problem to your manager:

  • The situation does not require immediate intervention (i.e. no laws broken, no lives at risk, no bank accounts being drained, etc.)
  • You’re waiting on additional information that will make the scope of the issue clear
  • You control the flow of information, and your audience won’t learn about the issue from other sources
  • It’s possible that the situation can be fixed soon, allowing you to communicate both the problem and solution together
  • You suspect there may be related problems lurking that should be disclosed together

Mackerel recipes

211102-0111 python defining own types for typing

After writing whatever: str or Path or whataver: Union[str, Path] for the N-th time I googled how to do this better. Well, 1

from typing import Union
from pathlib import Path

pathlike = Union[str, Path]

whatever: pathlike = some_function()

def f_paths(path_one: pathlike):

  1. What is the correct way in python to annotate a path with type hints? - Stack Overflow ↩︎

211102-1811 python pip and wheel

Python uninstalling requirements.txt

You can do python -m pip uninstall -r requirements.txt

python3 bdist_wheel errors

Errors with bdist_wheel missing as a command when installing python packages got fixed with the help of SO1, needed to do python3 -m pip install wheel


  1. Why is python setup.py saying invalid command ‘bdist_wheel’ on Travis CI? - Stack Overflow ↩︎

211101-2011 Git reset types

An incredibly clear explanation, copypasted from StackOverflow, about the flavours of git reset --xxx HEAD~1

In the simplest terms:

  • --soft: uncommit changes, changes are left staged (index).
  • --mixed (default): uncommit + unstage changes, changes are left in working tree.
  • --hard: uncommit + unstage + delete changes, nothing left.

211101-2111 bash - Find the size of all files of a certain type

From SO, to find the disk space taken by files with a certain extension/type:1

find ./photos/john_doe -type f -name '*.jpg' -exec du -ch {} + | grep total$

  1. Find the total size of certain files within a directory branch - Unix & Linux Stack Exchange ↩︎

211101-2211 NixOS and nix

I should really try this sometime. Having a reproducible OS install would make life much easier. On my radar a long time, but a person I was interviewing last week was the final drop I guess.

211101-2311 git push all local branches to remote or to different branch

From FreeCodeCamp:1

  • git branch shows all branches
  • git push --all pushes all local branches to remote.
  • git push origin some-branch:my-feature pushes the local branch some-branch to a remote branch called my-feature

  1. Git Push to Remote Branch – How to Push a Local Branch to Origin ↩︎

211028-1110 Python staticmethod vs classmethod

A @classmethod gets the class as first parameter, nice for constructors/factories etc. A @staticmethod doesn’t know anything about the class at all, and the only use it has is to put functions that logically belong to the class inside the class. 1

Additionally,


  1. python - Difference between staticmethod and classmethod - Stack Overflow ↩︎

211020-1410 ML starter kit resources website

ML Starter Kit

Contains books / resources about ML, from foundations to roadmaps / learning paths , “channels” (sites that regularly publish ML content), etc.

Really really impressive.

YAML Norway issues

Yaml 1.1 interprets the following strings as booleans, if unquoted: 1

 y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF

Related + YAML hate:


  1. Boolean Language-Independent Type for YAML™ Version 1.1 ↩︎

YAML Norway issues

Yaml 1.1 interprets the following strings as booleans, if unquoted: 1

 y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF

Related + YAML hate:


  1. Boolean Language-Independent Type for YAML™ Version 1.1 ↩︎

Салат с консервированным тунцом и фасольюDRAFT

Салат с консервированным тунцом и стручковой фасолью.🍅 ⠀ ✅КБЖУ:72/6/4/4 ⠀ Ингредиенты ▫️100 г стручковой фасоли ▫️30 г микса салатов ▫️150 г помидор ▫️1 болгарский перец ▫️3-4 шт. оливки ▫️100 г консервированного тунца ▫️1 отварное яйцо ▫️1 ст.л. оливкового масла (+ немного для обжарки) ⠀ Приготовление: ▫️Фасоль отварить в кипящей воде в течение 10 минут. Помидоры и перец нарезать. ▫️Перец обжарить на небольшом количестве масла в течение 5 минут. Оливки и очищенное яйцо нарезать ▫️Тунца размять вилкой. Все смешать, заправив оливковым маслом

ПП рецепты🍏 | #ппсалаты

Day 1021

Obsidian for zettelkasten-type notes

.. is probably my new obsession, along with getting it to play nicely with Hugo. It’s a closed non-open-source system but files are saved as markdown, has an awesome Android app - everything I’ve ever wanted except openness, basically.

So:

Template to create hugo-compatible front matter in Obsidian:

Templater1 is a community plugin for template stuff, but supports neat things like getting clipboard data, creating files, etc. Additionally supports automatically using templates when creating notes in a folder or in general and a lot of other excellent stuff.

This template gets run manually after I create and name a note. When I run it, it autogenerates Hugo front matter, gets the title from the filename, and puts the cursor in the first tag. The second tag is created from the folder name where the note is located, currently I defined two: it and rl.

---
title: "<% tp.file.title %>"
tags:
  - "zc"
  - "zc/<% tp.file.folder() %>"
  - "<% tp.file.cursor() %>"
fulldate: <% tp.date.now("YYYY-MM-DDTHH:MM:SSZZ") %>
date: <% tp.date.now("YYYY-MM-DD") %>
hidden: false
draft: true
---

Obsidian to Hugo Conversion

I looked at zoni/obsidian-export: Rust library and CLI to export an Obsidian vault to regular Markdown and khalednassar/obyde: A minimal tool to convert a standardly configured Obsidian vault to a Jekyll or Hugo blog., found the latter to be a bit clearer in how it handles assets etc. It requires a date in frontmatter in YYYY-MM-DD format, which I provided.


  1. Templater ↩︎

211018-1510 Python rounding behaviour

round() has weirdly unexpected behaviour that I’m ashamed I didn’t notice or know about:

if two multiples are equally close, rounding is done toward the even choice (so, for example, both round(0.5) and round(-0.5) are 0, and round(1.5) is 2) 1

So:

>>> round(1.5)
2
>>> round(2.5)
2
>>> round(3.5)
4

  1. Built-in Functions — Python 3.10.0 documentation ↩︎

211018-1610 TODO - Garden and DTBDRAFT

Things to be done:

  • Improved templating
  • Shouldn’t rely on /days, should be in the root
  • The submenu should be a list of pages that have the tag “menu” or something
  • Version control and sync with phone

Cooking - generalDRAFT

General

1 cup is 236ml C = (F-30)/2 conversion is a good approximation only for weather, not oven temperatures!

F to C conversion table1

C F
225 110
250 130
275 140
300 150
325 165
350 177
375 190
400 200
425 220
450 230
475 245
500 260
550 290

  1. Отава Ё - Ой, Дуся, ой, Маруся (казачья лезгинка) Otava Yo - (Cossack’s lezginka) - YouTube ↩︎

DaysDRAFT

Garden

This will be the place where I experiment with Obsidian as a potential to replace my current sets of scripts in use in the Diensttagebuch etc.

Even more than all other pages on https://serhii.net/* , this place should be considered work in progress and likely to explode/disappear at any time.

Interesting pages:

  • [[garden/rl/Traveling checklist]]

Lemon curd salmon recipeDRAFT

Basics: salmon, lemon curd, lemons, 15min in preheated oven Source: 5 ingredient Lemon Curd Salmon | Made It. Ate It. Loved It.

Ingredients

  • 1 salmon filet
  • 1 560g jar lemon curd
  • 1/4 cup (60ml) butter, sliced
  • 2 lemons (1 sliced)
  • salt and pepper

Time

~35min

(Preheat oven, 15min to prepare, 15min in oven.)

Steps

  1. Preheat oven to 175 C.
  2. Grab the salmon filet, take the salt and pepper and season it.
  3. Then take the jar of lemon curd and spread on about half of it, or until the salmon is covered.
  4. Then place the butter slices on top of the salmon.
  5. Then top with lemon slices. Take the other lemon and squeeze the juice on top of the lemon.
  6. Pop in the oven and bake for about 15-20 minutes.
  7. Once done, remove from oven and spread more lemon curd on top!

Video

САМЫЙ УНИВЕРСАЛЬНЫЙ «ПОХУДИТЕЛЬНЫЙ» САЛАТDRAFT

Ingredients

  • 1/2 баночки тунца в собственном соку (90-100 гр)
  • огурец
  • помидор
  • листья салата
  • зерненная горчица
  • соль, сухой чеснок

Steps

  • Все смешиваем, заправляем оливковым маслом

Day 1018

Python math.isclose() to check for “almost equal”

Had an issue with checking whether a sum of floats sums up to a number, remembering that python floats are ‘special’:

>>> 0.1 + 0.2
0.30000000000000004

Stack overflow1 told me about math.isclose(), works as you’d expect:

assert math.isclose(sum(floats), needed_sum)

  1. python - pytest: assert almost equal - Stack Overflow ↩︎

Day 1015

unittest skip test based on condition

From unittest documentation 1

class MyTestCase(unittest.TestCase):

    @unittest.skipIf(mylib.__version__ < (1, 3), "not supported in this library version")
    def test_format(self):
        # Tests that work for only a certain version of the library.
        pass

  1. unittest — Unit testing framework — Python 3.10.0 documentation ↩︎

Day 1009

Google Meet

You can minimize your own video, and then make the entire window much smaller!

Python strings formatting

Obvious, but: you can declare strings and format them in separate places!

constants.py:

my_string = "Hello my name is {0}"

other_file.py:

from constants import my_string
print(my_string.format("Serhii"))

Pycharm run current unittest binding

<C-S-F10> runs the unittest where the cursor is currently located. Or all of them if located anywhere else in the file.

TODO: set binding to do the same, but debugging.

python - run only some tests from a test suite

I wanted to run only some test files, all except the ones where I needed a GPU. Wrote this:

import subprocess

# Parts of filenames to exclude
large_tests = ['component', 'test_temp']

test_folder = Path(__file__).parent.absolute()
test_files = list(test_folder.glob("test_*.py"))
test_files = [x.name for x in test_files]

for l in large_tests:
  test_files = list(filter(lambda x: l not in x, test_files))

commands = ["python3", "-m", "unittest"] + test_files

subprocess.run(commands, cwd=test_folder)

Notes:

  • Thought this would be a security nightmare, but it’s not1 - unless shell=True is explicitly passed, no shell is called, ergo no shell-command-injection stuff is possible.
  • os.chdir() is nicely replaced by the cwd= parameter, much nicer than what I’d have done previously!

  1. subprocess — Subprocess management — Python 3.10.0 documentation ↩︎

Day 1008

Python typing annotating second-order functions

def my_function(other_function: Callable) -> Callable:
  return other_function

Pycharm run all unit tests in a folder

What I’d do as

cd tests
python3 -m unittest

in Pycharm is right-clicking on a directory in Project view and “Run unittests”

OOP principles

Open/Closed principle: you should be able to open a module/class to add stuff easily, but otherwise you shouldn’t need to touch it for existing stuff.

Python dir

Wrote a line like if dir is not None .., but dir is a builtin! It returns all the names in the current scope.

Pycharm debugging

You can add Watches, values that will be shown and tracked! Nice for debugging stuff that needs values that are deep in other variables

Python unittests run at the end of the class/module

  • class-level:

    • setUpClass(cls) gets called before tests from one class get run, not once per test
    • tearDownClass(cls) gets called before tests from one class get run, not once per test
    • Both need to be class methods, a la:1
        class Test(unittest.TestCase):
            @classmethod
            def setUpClass(cls):
                cls._connection = createExpensiveConnectionObject()
      
  • module-level

    • setUpModule(), tearDownModule()
    • should be implemented as normal functions

    Aaanad if you set any class variables, you can still access them as self.xxx from within the tests!

Python or in arguments

Neat thing seen in detectron default_argument_parser:

def argparser(epilog=None):
  ...
  x = epilog or "here's some text"

Where “here’s some text” is a long string that doesn’t really belong in the function signature.

A really nice pattern, much better than my usual

if x is None:
  x = ...

  1. unittest — Unit testing framework — Python 3.9.7 documentation ↩︎

Day 1007

vim open list of files from cli

vim -p `ag -l whatever`

opens each file returned by ag. (ag -l lists only the files with matches and nothing else)

Replacing jekyll-style highlight tags with standard markdown ones

In some posts I had code blocks like {% highlight html %} etc. The html/js got parsed, and some “here’s how to redirect using javascript” code got executed in the master page.

Here’s how I replaced all that syntax with the standard markdown one:

for f in `ag -l "endhighlight"`;
do cat $f | sed "s/{% highlight \(.*\) %}/\`\`\`\1/" | sed "s/{% endhighlight %}/\`\`\`/g" > $f;
done

Python dataclasses and classmethods

@dataclass
class MyClass:
  x: int = 4

@classmethod
def init_whatever(number: int)
  return cls(x=number)

Python exceptions and unittests

unittest’s self.assertRaisesRegex() is nice but couldn’t get it to work with my custom exception class.

with self.assertRaisesRegex(CustomException, "No trained model"):

It expects the message to be in e.args1. args also gets used by the Exception class for __str__() etc, so it’s a nice thing.

Set it up easily:

class CustomException(Exception):
    def __init__(self, detailed_message: str = None):
        if detailed_message:
          self.detailed_message = detailed_message
          self.args = (self.detailed_message, )

Catching python regex exceptions

try:
  re.search("DataLoader worker(.*is killed by signal: Bus error", text)
except re.error:
  whatever()

TODO I really like this regex tutorial: Regular Expressions: Regexes in Python (Part 2) – Real Python


  1. 8. Errors and Exceptions — Python 3.9.7 documentation ↩︎

Day 1006

Hugo indexes and layouts

I think that:

  • Placing an _index.md in the root of the section makes it listable with a list.html template.
  • Placing an index.md (no underscore!) makes that file’s content the real index of that section.

The best way to use a custom layout is to specify it explicitly in the front matter as layout: newlayout. For example for the custom list template in pages (formerly /ntb/pages), I put the layout file in ./layouts/ntb/ntblist.html and put in ./content/ntb/pages/_index.md’s front matter this:

title: "Pages"
[...]
layout: ntblist

Day 1003

Pycharm presentation mode and font size

Previously, I had to manually increase font sizes in Pycharm when presenting stuff in meeting, and couldn’t automate it.

Today I realized that I can change the script resolution to a lower one, giving the same results, and easily automatable through randr and a shell script!

Pycharm moving functions

“Right click -> Refactor” works not just for renaming files/folders, but also for moving functions to different files!

Miroboard moving

Holding <space> makes the mouse move the view, not the content

Logging in Python

logging — Logging facility for Python — Python 3.9.7 documentation

logger.exception() exists! Exception info is written as given by the exception handler.

Exceptions handling in Python

Was looking for a strategy to handle errors in a complex-ish applications, with logging, different levels etc.

  • Three options how to deal with exceptions:1

    • Swallow it quietly (handle it and keep running).
    • Do something like logging, but re-raise the same exception to let higher levels handle.
    • Raise a different exception instead of the original.
  • Defining your custom exception1

    class SuperError(Exception):
        def __init__(self, message):
            Exception.__init__(message)
            self.when = datetime.now()
    
    raise SuperError('Something went wrong')
    
  • Re-raising the same exception after handling it 1

    def invoke_function(func, *args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            print type(e)
            raise
    
  • Ways to clean stuff up in try..catch blocks:2

    • try: - execute this
    • except: execute this if there’s an exception
    • else: - execute if no exceptions
    • finally: - always run this code
  • Context managers

    • Alternative to finally, standard with ... syntax
  • Logging best practice1

    import logging
    logger = logging.getLogger()
    def f():
        try:
            flaky_func()
        except Exception:
            logger.exception()
            raise
    

    If you re-raise, make you sure you don’t log the same exception over and over again at different levels.1

    The simplest way to do it is to let all exceptions propagate (unless they can be handled confidently and swallowed earlier) and then do the logging close to the top level of your application/system.1

  • Error logger decorator for the above1

    def log_error(logger)
        def decorated(f):
            @functools.wraps(f)
            def wrapped(*args, **kwargs):
                try:
                    return f(*args, **kwargs)
                except Exception as e:
                    if logger:
                        logger.exception(e)
                    raise
            return wrapped
        return decorated
    

    And usage:

    import logging
    logger = logging.getLogger()
    
    @log_error(logger)
    def f():
        raise Exception('I am exceptional')
    
  • If there are multiple decorators, that one should be the immediate next one to the function! When I did it wrong, I got an exception (ha) about “‘staticmethod’ object is not callable”.

    The correct way is:

    @staticmethod
    @return_trainer_exception(logger=None)
    

  1. Professional Error Handling With Python ↩︎

  2. Python Exceptions: An Introduction – Real Python ↩︎

Day 1002

Cherry-picking commits from pycharm

Messed up merging/rebasing branches from branches from branches, but needed to merge literally a couple of commits.

So I created a clean branch from master. Then:

  • Check out the target branch, the one you’re cherry-picking to
  • Open the git log
  • Select the commits you want to cherry-pick, right click, “cherry-pick”
  • Done!

As usual, docs exist1 and are quite readable.

PEP8 max line width of 80 characters

… is the best thing since sliced bread, I was skeptical at first but makes editing code in multiple windows so much better!

Installing python3.8 on Ubuntu 18.04 LTS

  • Needed a third-party PPA2 that has all the newer python versions:
      sudo add-apt-repository ppa:deadsnakes/ppa
      sudo apt install python3.8
      sudo apt install python3.8-dev
    
  • Needed sudo apt-get install python3.8-venv3
  • Needing to reinstall all packages for it, haha.
    • Set up locally a venv38 for this; if I source venv38/bin/activate python3 becomes python3.8 by default.
  • Needed to install

python3.8-dev was added after an error I had4 when installing pycocotools, it didn’t find python.h when building.

Installing python locally

This describes the process well: Install python3.6+ for local user on remote machine without root access - ~/web/logs

The official documentation: 2. Using Python on Unix platforms — Python 3.9.7 documentation

Basically make altinstall is a safer version that doesn’t overwrite system-wide stuff:

make install can overwrite or masquerade the python3 binary. make altinstall is therefore recommended instead of make install since it only installs exec_prefix/bin/pythonversion.

TL;DR:

  • Download the source tarball
  • Then:
    ./configure --prefix=whatever
    make
    make altinstall
    
  • Add the prefix to $PATH:
    export PATH=$PATH:/data/sh/tools/python3.8/bin
    

Hugo auto-reload and CSS

Just now remembered that when doing CSS stuff it’s sometimes cached, and one needs to <Shift-R> or sth similar. Hugo’s automatic reloading reloads the content/templates/…, but not the CSS!

Explains a lot of what happened the last two days.

Hugo Templating

Copypasting from the docu5:

  • Parameters for functions are separated using spaces
  • Dot-notations for methods and fields ({{ .Params.bar }})
  • Things can be grouped via parentheses:
    • {{ if or (isset .Params "alt") (isset .Params "caption") }} Caption {{ end }}
  • A Single Statement Can be Split over Multiple Lines:
    {{ if or 
      (isset .Params "alt") 
      (isset .Params "caption")
    }}
    

Setting directory-specific settings in vim

Given that Hugo’s markdown considers code as part of a bullet-point if it’s indented two spaces more than the *-bulletpoint’s level, and that I have a tabwidth of 4 and tabs everywhere else and two spaces were a pain…

To apply settings only within a specific directory, add this to ~/.vimrc6:

autocmd BufNewFile,BufRead /home/me/ntb/* set tabstop=4 softtabstop=4 shiftwidth=4 expandtab foldmethod=marker

Notably, for me it didn’t work when the path contained a symlink, had to write it explicitly.

Another option from that SO questiont was placing a ~/.vimrc in that directory7, allowing vim to use it by default, and sourcing the usual global one from the first line. Has security implications, may lead to issues with paths/plugins, didn’t try it.

vim tabs and spaces and indentation settings

Looking for indentation stuff for the above lead me here: Tab settings in Vim. Summary: | by Ari Sweedler | Medium

It has this description, copying verbatim:

  • tabstop: display-only, how many spaces does one \t equal visually?
  • shiftwidth: how many spaces does one level of indentation equal? (shifting commands, formatting, behaviour).
  • softtabstop: how much whitespace to add/remove when pressing tab/backspace?
    • Disabled by default; if using tabs - we create as much whitespace as needed to get to the next tabstop
    • but when using spaces for indentation, we don’t want backspace to delete one space, then this is needed
  • expandtab: should pressing <Tab> on the keyboard create spaces or a tab character?

highlight indentation levels in vim, if indentation is done with spaces

Highlighting tab-indents is easy, and I had these settings for that:

set listchars=tab:\:\ 
set listchars+=trail:◦

For spaces it’s harder.

Tried the indentLine plugin8, due to it using the conceal setting I couldn’t see my json-quotes and _ underscores anymore. Setting conceallevel to 1 from 2 helped only for the latter. May get fixed by colorscheme/syntax files with less concealed stuff?

Setting let g:indentLine_concealcursor = '' (by default inc) helps - text is not concealed at all in the cursor line in any of the modes. I see all concealed text and don’t see the guides. I can kinda live with that.

In any case replacing the 's in json is ugly.

Then found this excellent SO answer. set cursorcolumn cursorline highlight the entire column/row where the cursor is. Which is why I want indentation highlighting 99% of the time!

With my newfound vim knowledge, added this to ~/.vimrc:

autocmd filetype python set cursorcolumn cursorline

But this didn’t satisfy me for the dtb and I kept looking.

Then I found vim-indent-guides9 that changes just the background color. Settings I ended up using:

let g:indent_guides_enable_on_vim_startup = 1
let g:indent_guides_auto_colors = 0
let g:indent_guides_start_level = 2
let g:indent_guides_guide_size = 4
" autocmd VimEnter,Colorscheme * :hi IndentGuidesOdd  guibg=darkgrey  ctermbg=233
autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=233

ctermbg=233is one of the darkest black-like vim colors, there’s a nice vim colors reference10 online.

At the end, wrapped everything related to DTB and indentst in one nice startup function:

fun! SetDTB()
	set tabstop=4  shiftwidth=2 expandtab 
	foldmethod=marker
	set nocursorline nocursorcolumn 
	let g:indent_guides_auto_colors = 0
	let g:indent_guides_start_level = 1
	let g:indent_guides_guide_size = 1
	autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=236
endfu

autocmd BufNewFile,BufRead /home/me/ntb/* :call SetDTB()

  1. Apply changes from one Git branch to another | PyCharm ↩︎

  2. New Python Versions : “deadsnakes” team ↩︎

  3. python - pyvenv not working because ensurepip is not available - Stack Overflow ↩︎

  4. make error under PythonAPI, python.h No such file or directory · Issue #180 · cocodataset/cocoapi ↩︎

  5. Introduction to Hugo Templating | Hugo↩︎

  6. Vim: apply settings on files in directory - Stack Overflow ↩︎

  7. Answer about local .vimrc in Vim: apply settings on files in directory - Stack Overflow ↩︎

  8. Yggdroot/indentLine: A vim plugin to display the indention levels with thin vertical lines ↩︎

  9. nathanaelkane/vim-indent-guides: A Vim plugin for visually displaying indent levels in code ↩︎

  10. 256 Colors - Cheat Sheet - Xterm, HEX, RGB, HSL ↩︎

Day 1001

1001th post in Hugo!

Set up Hugo for DTB and partly sth I’ll call NTB, which is non-work stuff.

So far Hugo is 110/10.

Old one for now is here.

Jekyll to Hugo

TODO:

  • Aliases/redirects from old posts to new ones (serhii.net/day123.html -> serhii.net/day123)
    • uglyurls: true in config does exactly this!
    • …but breaks lists/indexes somehow :(
  • Look through master file for formatting issues
  • Better black-background syntax highlighting if no language specified
    • Ideally make them indistinguishable from styled ones
    • And remove ghost ones like day 996
      • The problem was with my markdown syntax, apparently *I need a two space indentation from the * for it to be parsed correctly. Another reason to revisit my vim tab settings?
    • using '''text seems like a workaround:
      This is text
      No syntax highlighting
      
      And:
      This is text
      No syntax highlighting _at all_
      
  • Randomized footers
  • Set up Atom feed on home page
    • Or actually - I could move the entire website to Hugo, and have the index-index as a template and /dtb for the posts stuff?
  • Strikethrough
    • Markdown strikethrough is ~~strikethrough~~ 1
  • Fix code listings' width influencing the width of entire Day.

tree including hidden files

I love how intuitive it is - needed a dotfile in tree, tried tree -a, it worked.

Python unittest

setUp() and tearDown() methods in unittests get executed before/after each test method!

Unregistering Detectron2 datasets for unittests

The dictionary with the datasets is a global dictionary, which means that you can’t register_coco_instances() in separate unittests in the same file!

This worked:

if Constants.TRAIN_DATASET_NAME in MetadataCatalog.data:
    MetadataCatalog.remove(Constants.TRAIN_DATASET_NAME)
    MetadataCatalog.remove(Constants.TEST_DATASET_NAME)
    DatasetCatalog.remove(Constants.TRAIN_DATASET_NAME)
    DatasetCatalog.remove(Constants.TEST_DATASET_NAME)

Pycharm / Intellij idea visual guides for character limit

Through IDE settings one can configure whether one or multiple visual guides are shown, and the actual number of characters is configured through Settings -> Code Style.

Random

Jupyter notebooks + RISE + Reveal.js + a makefile: cornell-cs5785-2021-applied-ml/Makefile at main · kuleshov/cornell-cs5785-2021-applied-ml

TODO - Git - squashing multiple commits into one

Squash commits into one with Git - Internal Pointers (link by SO):

# Merge the last 7 commits into one
git rebase --interactive HEAD~[7]
# Merge the commits from that commit hash
git rebase --interactive 6394dc

In the latest one, the commit hash is “the hash of the commit just before the first one you want to rewrite from.”

Practically, assuming I want to squash together the a ones, I’d do git rebase --interactive B as it’s the one immediately following the ones I need.

commit a1 (latest/newest)
commit a2
commit a3
commit B
commit C

When actually doing the squashing, set squash in front of the commit lines to squash. In the next screen, leave only the commit message(s) needed.

I love how it uses vim for this! Very interesting way to do an interface.


  1. Extended Syntax | Markdown Guide ↩︎

Day 1000

Python PEP8 / black / flake8 / style

flake8 file.py shows issues;

black file.py applies black. black --line-length=79 file.py applies the line length as per PEP8.

Pycharm uses 119 characters as limit, coming from Intellij I think; officially PEP8 recommends 79.

German / Words

Blau sein = be drunk (heard at work)

Day 997

Hugo the static site generator

My blog takes minutes to be generated, this DTB is not far from it either. I heard Hugo is fast, and I dislike most of what’s on my blog, the logical thing seems to burn it to the ground and start from zero using Hugo.

cd themes
git submodule add https://github.com/chollinger93/ink-free
cd ..
echo theme = \"ink-free\" >> config.toml
  • Creating a post:
hugo new posts/my-first-post.md

puts the file in ./content/posts/my-first-post.md

  • Building:
    • Starting the local server: hugo server -D
    • REALLY fast, and reloaded live in my browser!
    • Building the site: hugo -D
  • Configs
    • config.toml supports #comments
    • con-fig and con-tent in the same folder make my tab completion sad.
    • Configure Hugo | Hugo
    • It supports .yaml, .json and .toml configs and config directories!
  • Directory structure: Directory Structure | Hugo
    • Site structure in inferred by directories: Content Sections | Hugo
      • They still need to be added to config to be in the menu
      • Nevertheless accessible when typing the URL directly
      • A subdirectory is a navigable section only if it has an _index.md
    • hugo new content/pages/one/two/test-page.md
      • The command only from parent of config
      • It generates the boilerplate, I don’t need to write a script for that! It even gets the title from the filename!
      • If there’s an archetype with the same name it’ll use that!
  • Writing / content creation

Day 996

Python typing cheatsheet & other stuff

Nice cheatsheet, not mypy-specific: Type hints cheat sheet (Python 3) — Mypy 0.910 documentation

Otherwise:

  • Functions that may return None:
    • x = 1 if True else None, x would be Optional[int]
  • Iterables / Sequences:
    • Iterable is anything usable inside a for
    • Sequence is anything that supports len()
    • For example:
      def f(ints: Iterable[int]) -> List[str]:
      return [str(x) for x in ints]
      

python unittests through CLI

From docu 1:

python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method

When I’m in the directory with the test_xy.py files, running python3 -m unittest runs all of them. I can also do python3 -m unittest test_xy for that file, and python3 -m unittest test_xy.TestXY.test_specific_thing.

Debugging python from CLI through breakpoints

Found this, and it’s freaking awesome: Debugging by starting a REPL at a breakpoint is fun

Sample from there:

def make_request():
    result = requests.get("https://google.com")
    import ipdb; ipdb.set_trace()

There’s the default pdb, there’s ipdb that has to be installed.

Adding

import ipdb; ipdb.set_trace()

anywhere in the code launches a typical debug window that can be used to look into the vars etc.

Just used this for the first time to debug a python program that was running on a remote server and failing, but not locally.

SO much better than print(f"XXX {i}") and friends!

Nice tutorial about its usage: Better Python Debugging With IPDB

  • n - next line in current method (=“step over”)
  • s - next line of executable code anywhere (=“step into”)
  • c - continue till next breakpoint
  • r - continue till function returns (would be nice to learn how to do this in pycharm btw!)
  • a - args - print arguments current function received
  • b - adds breakpoint to locations
    • b filename.py:234
    • b <function>
    • b 123 - line in current file

Full documentation here: 26.2. pdb — The Python Debugger — Python 2.7.18 documentation

Python serializing Enums by declaring them as subclass of str

My main issue with Enum classes was that serialization is weird, especially if you’re dumping parameters. Tried again, found this: python - Serialising an Enum member to JSON - Stack Overflow

TL;DR class EnumDerivedClass(str, Enum)

import json
from enum import Enum

class LogLevel(str, Enum):
    DEBUG = 'DEBUG'
    INFO = 'INFO'

print(LogLevel.DEBUG)
print(json.dumps(LogLevel.DEBUG))
print(json.loads('"DEBUG"'))
print(LogLevel('DEBUG'))

will output

LogLevel.DEBUG
"DEBUG"
DEBUG
LogLevel.DEBUG

Google Presentations work in progress slides

“Folie überspringen” is a much better way to do what I did with setting a yellow background color - easy to see and worst case scenario it’ll just get skipped

Tensorboard and no data because wrong input folder

If you run tensorboard on a non-existing folder, you’ll get no feedback about it anywhere?.. No data on Tensorboard itself, nothing useful in CLI.


  1. unittest — Unit testing framework — Python 3.9.7 documentation ↩︎

Day 995

Pycharm / Intellij idea local history - for files and directories!

After some ill-fated undoing of commits, couldn’t find the work of an hour or so.

Guess what: Using Local History to Restore Code Fragments or Deleted Files | The IntelliJ IDEA Blog

I knew about local history for a file, but you can do the same for a directory, through its right-click menu in the Projects view!

Day 993

Nvidia GPU/eGPU drivers blues

I already thought I had set up nvidia-smi and friends (Day 850 | Diensttagebuch (Work journal)), then didn’t use it for months, now when I tried it didn’t work anymore, nvidia-smi said “No devices found”

boltctl showed the device as connected and authorized, prime-select said nvidia was selected, modprobe showed that the correct drivers were used and dkms status had said the correct drivers were installed.

(11:53:23/10181)~/$ dkms status
nvidia, 460.73.01, 5.4.0-73-generic, x86_64: installed
nvidia, 460.73.01, 5.4.0-74-generic, x86_64: installed

(11:53:49/10182)~/$ boltctl
[snip]
 ● Lenovo ThinkPad Thunderbolt 3 Dock #2
   ├─ type:          peripheral
   ├─ name:          ThinkPad Thunderbolt 3 Dock
   ├─ vendor:        Lenovo
   ├─ uuid:          xxx
   ├─ status:        authorized
   │  ├─ domain:     domain0
   │  └─ authflags:  none
   ├─ authorized:    Mo 20 Sep 2021 09:41:16 UTC
   ├─ connected:     Mo 20 Sep 2021 09:41:16 UTC
   └─ stored:        no

 ● GIGABYTE GV-N1070IXEB-8GD
   ├─ type:          peripheral
   ├─ name:          GV-N1070IXEB-8GD
   ├─ vendor:        GIGABYTE
   ├─ uuid:          xxx
   ├─ status:        authorized
   │  ├─ domain:     domain0
   │  └─ authflags:  none
   ├─ authorized:    Mo 20 Sep 2021 09:42:35 UTC
   ├─ connected:     Mo 20 Sep 2021 09:42:35 UTC
   └─ stored:        Mo 20 Sep 2021 09:31:09 UTC
      ├─ policy:     manual
      └─ key:        no

(11:54:54/10188)~/$ lsmod
Module                  Size  Used by
nvidia_uvm           1015808  0
nvidia_drm             57344  1
nvidia_modeset       1228800  1 nvidia_drm
nvidia              34123776  17 nvidia_uvm,nvidia_modeset

(11:55:54/10192)~/$ sudo prime-select query
nvidia

What didn’t work:

  • prime-select cycling to default and then back to nvidia and rebooting
  • power-cycling the CPU
  • Connecting it directly, not through the dock, exact same setup I had in when it was working (link above)

What worked:

  • Honestly no idea
  • logging into gnome, opening the driver config window, logging back into i3, rebooting?…

Offtopic, when I was googling these issues I found my own serhii.net link above on the first page of Google for the key '“nvidia-smi “no devices were found” authorized', which is both nice and sad at the same time :)

EDIT: the next morning it didn’t work again. None of the same magic steps in all possible orders. I think it might be an issue with the eGPU or dock or something of that level. The best way to check this would be to do the nuclear option, uninstall all drivers, and install from the beginning, but I think my monthly quota of GPU stuff is full five times over now.

Diensttagebuch / Meta

We’re on day 993 (!) of Diensttagebuch! Freaking awesome.

python pip “advanced” requirements.txt creation

Was creating a requirements.txt for detectron2, official install instructions were:

python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html

Answer specificalyl about this: python - How to format requirements.txt when package source is from specific websites? - Stack Overflow:

requirements.txt format is:

[[--option]...]
<requirement specifier> [; markers] [[--option]...]
<archive url/path>
[-e] <local project path>
[-e] <vcs project url>

<requirements specifier> is:

SomeProject
SomeProject == 1.3
SomeProject >=1.2,<2.0
SomeProject[foo, bar]
SomeProject~=1.4.2

The –option (such as the -f/–find-links) is the same as the pip install options you would use if you were doing pip install from the command line.

Therefore, in requirements.txt it ended up literally as this:

--find-links https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html detectron2

And by the way, detectron2’s own requirements.txt demonstrates nicely part of the above.

My own requirements.txt for CUDA 11.1:

opencv-python==4.2.0.32

# torch 1.9 for cuda 10.2 (for this config https://pytorch.org/get-started/locally/ has no versions in the command
# getting both exact versions from pip freeze
-f https://download.pytorch.org/whl/torch_stable.html
torch==1.9.0+cu111
torchvision==0.10.0+cu111
#torch==1.7.1
#torchvision==0.8.2

# python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html
-f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html
detectron2

grep/ag

Best part about ag is that I don’t need to escape anything with its default settings:

pip freeze | ag "(detectron|torch)"
pip freeze | grep "\(detectron\|torch\)"

pycharm test “teamcity” output bug

Suddenly stopped getting readable output. Fix is to add the env variable JB_DISABLE_BUFFERING, without any value, to the env of the test. teamcity - no output in console for unittests in pycharm 2017 - Stack Overflow

Day 989

Detectron2 parameters train/eval/checkpoint etc

The documentation about default confg covers all the parameters' meanings and can be used as reference for that! detectron2.config — detectron2 0.5 documentation

And me dreaming up cfg.MODEL.CHECKPOINT_PERIOD is exactly what they wanted to avoid by disallowing the creation of new keys.

Highlights:

# Number of images per batch across all machines. This is also the number
# of training images per step (i.e. per iteration). 
_C.SOLVER.IMS_PER_BATCH = 16

Phone disk space & Telegram cache

For the second time, discovered that Telegram Cache takes 40gb of disk space.

In the phone’s own menus related to disk space, this was shown as “Pictures” taking 40gb, not the Telegram app and its cache. But this info is exposed through Telegram’s own menus.

Day 988

timewarrior track and :fill

Who knew you could combine commands! This is how you start tracking tag1,tag2 starting from the end of the previous span:

$ w track :fill tag1,tag2

Backfilled to 2021-09-15T12:21:41
Tracking "tag1,tag2"
  Started 2021-09-15T12:21:41
  Current               23:47
  Total               0:02:06

Running DUC with sshfs (excluding files and filesystems)

TL;DR:

duc index ~/ --fs-exclude fuse.sshfs

duc is about disk space, before running it the index should be built/updated. Usually similar to duc index ~/.

If I have a sshfs mounted somewhere, the process never ends as it tries to index the folder where it’s mounted.

Found some solutions:

  • To exclude entire filesystems, duc index ~/ --fs-exclude fuse.sshfs
    • According to the man page, this would be a comma-separated list of filesystems as found in fstab, like ext3,ext4.
    • My /etc/fstab didn’t have the sshfs filesystem, but mount called it fuse.sshfs and this worked!
  • To exclude individual files, duc index ~/ -e "*somefilename*"
    • doesn’t seem to work with folders in all variations I could think of (*folder\/file* etc).
    • So no way to exclude a folder? Except using its name and praying no other folders share it

Bonus: -p shows progress during indexing.

Now I have a new alias in ~/.zshrc:

ducindex() {
	duc index "$1" -p --fs-exclude fuse.sshfs 
}

cdd CLI alias for CD-ing to directory containing a file

I copypaste a lot the locations of the files from pycharm/intellij to run them from CLI or something similar. Easiest way, because they are focused and I don’t need to focus on the files/project view for that. I can’t find an Action in pycharm/intellij to copypaste only the directory.

Yet another alias for today:

cdd() {
	$(dirname "$1")
}

dirname gives the directory, dirname .. | cd and dirname ... | xargs cd don’t work (TODO - why?), so I’m using the zsh thing about “cd to the directory if it’s in a command by itself”.

Now cdd /home/me/wahtever/test.py takes me to /home/me/whatever/ which will saved tens of seconds per year!

Concatenating/splitting tiffs

Of course tiffsplit1 has a sister tiffcp! Ubuntu Manpage: tiffcp - copy (and possibly convert) a TIFF file

Concatenate N pages into a result.tif:

tiffcp xaaa.tif xaab.tif xabr.tif result.tif

pycharm highlights comments if they’re immediately below a TODO one and indented

# TODO - this is highlighted yellow
# And this is not

# ... BUT!

# TODO - this is highlighted yellow
#  This too, because it's indented one space and logically belongs to the comment above!

Random / vim / TODO

I often do <CR>ddkkp or d$kp as a reverse-enter, moving what’s to the right of the cursor on the line above the current one. I’m sure something like this already exists in vim.

Detectron2 and Yacs / YAML config / CfgNode; allow adding new keys

Detectron’s Yacs has a github repo with documentation and examples, much better than detectron’s own: rbgirshick/yacs: YACS – Yet Another Configuration System

This works:

comp_cfg.set_new_allowed(True)
comp_cfg['key'] = 'value'

Interesting bit about why it’s not like this by default:

We typically don’t use this so that typo in config file can be noticed. 2

Additionally, this is set per leaf, not per config - you can allow adding stuff to the root but not to its existing children.

And, still, even with comp_cfg.set_new_allowed(True), why can’t I merge_from_list etc. for non-existing keys? (TODO)

Detectron’s logger and log.txt

log.txt is nice and colorful on CLI, I don’t remember how to let vim interpret the CLI colors but less log.txt works magnificently.

cfg.merge_from_file() doesn’t work with new keys · Issue #2082 · facebookresearch/detectron2


  1. Ubuntu Manpage: tiffsplit - split a multi-image TIFF into single-image TIFF files ↩︎

  2.  ↩︎

Day 986

Write full screen output/buffer to a file

If you are inside a screen, and need to write the entire contents to a file (not just the ones currently visible), this will work:

<C-a> :hardcopy -h <filename>.

Day 974

Random / language / English

In the context of a raised hand in google Hangouts meeting: “Do you have a question or an opinion?” (heard at work)

Intellij idea / Pycharm presentation mode

…TIL at work in a remote meeting. Makes the window with the code full-screen, hides the other windows, and increases the font size. Neat!

Day 972

Python itertools.count()

Yet another chapter of “python stdlib implementing most things I need better than me”, to create an infinite iterator itertools.count() is better than stuff like iter(range(100500)) (from AA’s comment in a PR)

Day 968

Detectron2, COCO datasets and negative examples

Detectron2 in its default dataloader filters images not containing any annotations1 because tradition; can be disabled by with

cfg.DATALOADER.FILTER_EMPTY_ANNOTATIONS=False

  1. Does this repo’s implementation of maskrcnn work with negative samples? · Issue #80 · facebookresearch/detectron2 ↩︎

Day 965

Ethernet device ‘not managed’ in network-manager

Couldn’t use ethernet because the device was ‘not managed’ according to nm-applet.

Neither

sudo nmcli dev set enp0s31f6 managed yes

nor changing managed=false to managed=true in /etc/NetworkManager/NetworkManager.conf helped (after the usual service restarts).

But creating an this empty file did:

sudo touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf

Python temporary directories

Memory lapse on my side, I thought tempfile.gettempdir() returned a random temporary directory I can use. Nope, it returns the absolute address of /tmp or its equivalent on that platform. I was thinking about tempfile.gettempdir(). There are also tempfile.TemporaryDirectory(), which gets automatically removed after the context ends or the object is deleted.

It’s the kind of things I’m afraid of in shell scripts, where manually deleting a temporary directory could remove more than needed.

As usual, the python docu on topic 1 is a good thing to read.

Python pathlib removing directory tree

There’s no way to remove a directory with all its contents recursively using pathlib. 2

pathlib.rmdir() removes empty directories, pathlib.unlink() removes files.

The way to do this is external libs, a la shutil.rmtree().

Very very weird design decision, as removing stuff is in no way an uncommon operation.

But a recursive pathlib solution exists, from same StackOverflow answer:

from pathlib import Path

def rmdir(directory):
    directory = Path(directory)
    for item in directory.iterdir():
        if item.is_dir():
            rmdir(item)
        else:
            item.unlink()
    directory.rmdir()

rmdir(Path("dir/"))

Python serialization of dataclass, datetime, numpy and stuff

orjson looks interesting: Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy | PythonRepo


  1. tempfile — Generate temporary files and directories — Python 3.9.6 documentation ↩︎

  2. directory - Deleting folders in python recursively - Stack Overflow ↩︎

Day 961

Pycharm Code Inspection

Can be run on an entire folder on right click -> “Inspect Code”

qutebrowser

Day 960

Changes in colorschemes/themes for low battery / low brightness / dark contexts

When coding in a plane and then on a bus did some slight changes, some are useful:

  • Intellij / pycharm:
    • “Darcula” / “High contrast” themes, both for editor and for IDE, are really nice when doing stuff in the dark
      • “High contrast” especially when using low screen brightness
    • When you change the IDE theme, you get a prompt to change the editor theme too
  • kitty / CLI
    • Increased font size to 13 and made it bold - made stuff much easier to see, especially the bold part.
    • Keeping the text bold by default from now on!
font_family      FiraCode-Bold
font_size 12.0
  • Was unable to get solarized CSS files working in qutebrowser for any website I tried to

If I’ll be on the road more often, I’ll create this as a mode or something - bold bigger text, different IDE colorschemes, etc.

English / phrasse

“Octopus mode” for emergency-multitasking-stuff - heard at work (J.)

Day 943

CSS selectors based on attributes

Was redesigning my website, looked if there’s a smarter way to color links based on whether they are internal/external than manually adding classes to them. Well there is: Attribute selectors - CSS: Cascading Style Sheets | MDN

Attributes can be parsed based on prefixes, suffixes, containing something, belonging to a predefined list etc.

Full list: CSS selectors - CSS: Cascading Style Sheets | MDN

Day 942

Telegram desktop shortcuts (especially for strikethrough text)

Random list from the internet: Telegram Desktop Keyboard Shortcuts (hotkeys)

Here interesting is <C-S-x> for strikethrough text. The others there are all mostly useful.

Random

Would be neat to add some simple javascripts to the Checklists | Diensttagebuch, so that when I click each <li> it’ll become strikethrough-d. I’d be something a la document.querySelectorAll("li") + somethingsomethingOnClicksomething.

javascript - Change CSS properties on click - Stack Overflow, or whatever. Filling this as “todo” for some infinite time in the future. Likely not worth spending time on, as I nether am planning to travel too much, nor want to learn more about javascript.

It kept showing a “Thesis” link in the header, I couldn’t understand where from - well, I had a file called \, prolly a vim artifact, which was a copy of the thesis.md I’d been blaming. Removing \ removed the link. This also breaks my assumption that jekyll will ignore any non-md non-html files, noted.

Jekyll blues - unpublished posts staying uploaded

published: false in the front matter should’ve made the post disappear, but reloading it I could see it was still there. Then I noticed it did disappear from the category listings.

The issue was my use of rsync, a line I had copypasted a long time ago:

rsync -av _site/ me@server:/whatever --progress --update

It uploads incrementally only the changed files. No one said anything about deleting the deleted ones! Jekyll didn’t generate pages for those posts, but the ones on the server stayed there.

Not quite sure whether a fix is needed, for now just removed the directory from the server.

Day 940

Fastmail calendar

Has nice keyboard shortcuts, viewable with ?. Heavily vim-inspired

Day 930

Notes about a presentation about privacy

Deleted as they were not interesting/relevant anymore, but one of these days I’ll post my final (Russian-language) presentation somewhere here.

Day 924

Pycharm/intellij debugging adding watchers

You can add things like someObject.someFunction() and basically any python code there! And it starts getting evaluated immediately after adding, even without stepping through or anything similar! This will save me a lot of “Eval code” - whose last remaining purpose can then be .. is “exploratory debugging” a thing?

Pycharm/intellij “Go back”

There’s a “Go back” action, <C-A-Left> is the default mapping on my installation - does what it says on the box. Handy for going back after looking at the implementation of something etc etc etc. Can’t find it in the ideavim actionlist though :( Though found <C-O> to jump to the last edited line which is very handy too:

 * |CTRL-O|               {@link com.maddyhome.idea.vim.action.motion.mark.MotionJumpPreviousAction}

Life keeps telling me to learn the tools I use daily, to read the entire help/manual etc - maybe one day I’ll learn to do this.

Pycharm / intellij refactoring

If you refactor a loop variable, such as for t in ..., if you choose to replace strings in comments, it might replace that letter outside tokens - the “t” in “won’t”, for example. (Not that clicking “Refactor” without looking at the suggestions is ever a good idea).

Day 923

Python imports paths handling

Object-Detection-Metrics/_init_paths.py at master · rafaelpadilla/Object-Detection-Metrics doesn’t use a main function in the files it runs, but has this neat snippet to add the library to PATH. TODO - at which point does this file get run and using what mechanism?

Day 920

qutebrowser undo last closed tab OR WINDOW

Add :undo –window by toofar · Pull Request #4807 · qutebrowser/qutebrowser adds this ability, mapped to U by default. Works for windows!

qutebrowser reopen all tabs and windows on startup

In general with autosave set, if I’m disciplined enough to close it with :quit or something mapped to it, it should reopen all of them.

Object detection metrics blues

So, again:

  • AP is Average Precision, basically area of the PR curve.
  • mAP is Mean Average Precision, so additionally averaged over classes and IoU thresholds depending on context (according to my reading of the COCO rules).

Day 915

Daily/weekly/… cron jobs

Adding the files to /etc/cron.hourly/daily/weekly/… makes them executed at least once a X. Better than standard way for instances where the computer can be turned off during the planned time, and then it won’t execute - the way above makes sure it will.

Day 913

jq-like tool for CSV

Miller (mlr) is a tool for doing stuff to csvs like jq is for jsqn: Quick examples — Miller 5.10.2 documentation

Day 909

Python formatted strings for fun and profit

cocoapi/pycocoDemo.ipynb at master · cocodataset/cocoapi has a nice example of a use case that’s not printlns:

dataDir='..'
dataType='val2017'
annFile='{}/annotations/instances_{}.json'.format(dataDir,dataType)

Nested tqdm loops and pycharm

Nothing was working, neither tqdm nor atpbar, till I used “emulate terminal” in the running config. As soon as I did all bars started working!

Nested loops - for tqdm, nothing needed except just calling it twice. The inner loop, tqdm(iterator, leave=False) removes the 100% completed inner bar and restarts from 0, so only two bars are seen at the same time.

atpbar (alphatwirl/atpbar: Progress bars for threading and multiprocessing tasks on terminal and Jupyter Notebook) is basically like tqdm. Can’t find an option similar to leave=True (though didn’t look), and output looks juuust a bit nicer than vanilla tqdm.

Day 905

Estimate internet connection speed from CLI

Since speedtest-cli is dead, this is an option that works:

curl -o /dev/null http://speedtest-blr1.digitalocean.com/100mb.test

Run vim without any config

vim -u NONE. vim -u filenaem reads only that filename as .vimrc, NONE is a key to not use anything.

Day 899

vim magic / nomagic / verymagic

Finally decided to undertand this part: Vim documentation: pattern

  • \m is magic, \M is nomagic. \m/magic is the default.
  • \v is verymagic, \V is very nomagic

Handy table from the documentation:

Examples:
after:	  \v	   \m	    \M	     \V		matches 
		'magic' 'nomagic'
	  $	   $	    $	     \$		matches end-of-line
	  .	   .	    \.	     \.		matches any character
	  *	   *	    \*	     \*		any number of the previous atom
	  ()	   \(\)     \(\)     \(\)	grouping into an atom
	  |	   \|	    \|	     \|		separating alternatives
	  \a	   \a	    \a	     \a		alphabetic character
	  \\	   \\	    \\	     \\		literal backslash
	  \.	   \.	    .	     .		literal dot
	  \{	   {	    {	     {		literal '{'
	  a	   a	    a	     a		literal 'a'

Practically:

  • \v/verymagic - almost everything has a special meaning (numbers, letters and _ are the only ones parsed as-is)
  • \V/verynomagic - almost nothing has a special meaning, everything interpreted as-is EXCEPT \

A Vim Guide for Adept Users has these nice tips that I’ll stick to:

My advice in this madness: remember that very magic will allow you to use every regex metacharacter without escaping them, and that very nomagic oblige you to escape these metacharacters to use them.

and

I propose this simple rule:

  • When you need a regex, use “very magic” by adding \v before your pattern.
  • When you don’t need a regex, use “very nomagic” by adding \V before your pattern.

It also has this nice list:

\s or [:blank:] - whitespace characters.
[A-Z] or \u or [:upper:] - Uppercase.
[a-z] or \l or [:lower:] - Lowercase.
[0-9] or \d or [:digit:] - Digits.
\_ - Character class with end of line included.

Day 898

linux pkill

pkill aw- kills all processes whose name starts with aw-!

borg backup & rsync.net

rsync.net is a nice no-nonsense offering. They have special prices for borg backups: Cloud Storage for Offsite Backups - borg support

Blog post about setting it up: Remote Backups with Borg | The Cucurbit Developer

rsync.net itself has nice documetation about a lot of stuff: rsync.net Cloud Storage for Offsite Backups

Day 895

timewarrior :fill

:fill works not just for moving stuff, but also tracking!

If I tracked A from 11:00 to 11:23 and now it’s 11:30, I can do timew track 2min B :fill - it will create B from the end of the previous one until now, so 11:24 - 11:30.

<C-R> gets vi-mode into search mode, after returning to Normal mode n/N work just as expected to do a case-insensitive search of similar things in history

Choose default google account

How to Change Your Default Google Account on Mac or PC says that the first one I log into will be the default one.

CLI Dashboards

iptables / webmin

Webmin is cool and allows to move iptables rules!

wireguard/pihole docker

Title of the year award goes to IAmStoxe/wirehole: WireHole is a combination of WireGuard, Pi-hole, and Unbound in a docker-compose project with the intent of enabling users to quickly and easily create a personally managed full or split-tunnel WireGuard VPN with ad blocking capabilities thanks to Pi-hole, and DNS caching, additional privacy options, and upstream providers via Unbound.

Day 892

Intellij marking folders as roots

A top-level folder can be excluded, but any of the folders inside it can be marked as something else and that will override the parent! Very sensible decision actually, when I think about it

vim don’t clean clipboard buffer / + register when closing

From SO:1

autocmd VimLeave * call system("xclip -selection clipboard -i", getreg('+'))

Here vim’s system() command is interesting:

If you pass a second argument like this, Vim will write it to a temporary file and pipe it into the command on standard input.2

In any case, I should really write some alias to be able to use xclip and friends by passing parameters to them, not piping stuff - makes any kind of scripting with them much harder.

And to finish, Learn Vimscript the Hard Way seems to be still an excellent introduction to vim itself, even without the scripting part.

ag/grep output only capturing groups

This3 describes how to get ag to output not the match, but only a specific capturing group inside it:

ag -o 'https://\K.*?(?=")'

It uses PCRE features to remove stuff from before and from after the match:

  • \K resets the match start
  • (?=") sets the end to " - here, " is what should be after the match, but will not be included in it.

PCRE

Related is Learn PCRE in Y Minutes. PC in PCRE stands for “Perl Compatible”.

PCRE can be enabled in grep by doing grep -P, and it’s the default in ag.


  1. Prevent Vim from clearing the clipboard on exit - Stack Overflow ↩︎

  2. External Commands / Learn Vimscript the Hard Way ↩︎

  3. Print match only · Issue #400 · ggreer/the_silver_searcher ↩︎

Day 889

General DVC notes

  • Access:
    • Can directly get stuff from a repo when not inside a dvc project environment
      • Such as from within ML or code
      • Git repo has to be accessible ofc
    • DVC import - same as above, but also gets the metadata
      • Needs to be inside a DVC repo
        • Or have to do git init & dvc init first
    • Python bindings exist
  • Stages:
    • Nice and neat
    • parameters.yaml
    • See parametrization below for maybe easier ways to pass parameters
    • Otherwise you just have your script read parameters.yaml, and version parameters.yaml too

DVC parametrization

Parametrization · iterative/dvc Wiki is an experimental feature.

Allows to call parameters directly, such as:

stages:
  build:
    foreach: ${models}
    do:
      cmd: >- 
          python script.py
          --out ${item.filename}
          --thresh ${item.thresh}
      outs:
          - ${item.filename}

as opposed to getting your program to read parameters.yaml

Ipset ipv6 ranges; online subnet ip calculators

IPSet set structures: wiki.ipfire.org - IPset To create an ipv6 ipset that supports domain ranges, we need the hash:net one:

ipset create my6 hash:net family inet6

Nice subnet calculators:

iptables doesn’t do ipv6, but ip6tables does, seems to be installed by default along with vanilla iptables. Commands seem to be identical.

Iptables persistent

  • iptables-save > some_output_file to save them to a file (this alone doesn’t make it persist reboots)
  • The packageiptables-persistent does what is says on the label,1 for rules being saved in:
    • /etc/iptables/rules.v4
    • /etc/iptables/rules.v6

Ipset save and restore

ipset save > output_file
ipset save -f output_file

ipset restore -f output_file
ipset restore < output_file

The output files it generates seem to be the exact commands without the leading ipset ?

iptables and ipset persistence on yunohost

Looked into yunohost’s recommendations, there’s a best practice.2 Created a shell script that does ipset restore -f file and then runs the iptables commands, put it into /etc/yunohost/hooks.d/post_iptable_rules/99-specific_rules. Survived a reboot, mission accomplished.

mktemp for temporary files

> mktemp /tmp/somescript.XXXX
/tmp/somescript.6Zxi

mktemp creates random files with a set format, replacing the XXX with random characters, and returns the filename (+ can also create directories). Cool!


  1. Saving Iptables Firewall Rules Permanently - Thomas-Krenn-Wiki ↩︎

  2. Best practice to add custom IPtables? - Discuss - YunoHost Forum ↩︎

Day 888

Python env variables

theskumar/python-dotenv: Get and set values in your .env file in local and production servers.

duc for visualizing disk space

Duc: Dude, where are my bytes! - both GUI and cli interface. Love it!

bash - running multiple processes in parallel

#!/bin/bash
run_command(){
	echo "The thing that will be run in parallel"
}

for i in {1..20}
do
	run_command $i &
done
 

Day 883

Awesome Quantified Self

What do I need?

  • Something self-hosted to:
  • … transparently and seamlessly track stuff, kinda like android Nomie in the good old days, but with web and android support
  • … easily send/receive stuff using an API for my own visualizations

Options:

Random:

  • Would be nice if somehow the TOREADs from DTB got parsed, my added links from wallaby got parsed, all would live on serhii.net/f/? ..or somewhere else
  • How would that play with morning/evening pages, weekly reviews, checklists? They’d be their own data source to..?

Friends

JacobEvelyn/friends: Spend time with the people you care about. Introvert-tested. Extrovert-approved. is really nice!

> friends add activity three days ago: Some activity three days ago                                                      <<<
Activity added: "2021-05-30: Some activity three days ago"

# also works:
> friends list activities --since="two month ago"

As with taskwarrior, things can get arbitrarily shortened as long as they remain unique! friends a ac "some activity" (you can add both an activity and an alias)

Firefox for Android - using the old extensions! (And Fennec)

Found this: How to use collections on addons.mozilla.org | Firefox Help

TL;DR create an extension collection on Firefox’s website, then from Fennec or Firefox Nightly they can be installed! Wooooohooo!

Also TIL about Fennec - seems like a Firefox fork without features that are ‘considered harmful’

Taskwarrior logging an already completed task

task log adds a task and sets its status to completed! 1

As a bonus, tasks that don’t have a specific tag are task -notthistag list

Git add vim swap files to .gitignore

To add all the swapfiles generated by vim (.swp, .swo, etc) to gitignore:2

.*.sw*

Here’s also interesting Github’s own .gitignore for vim files: gitignore/Vim.gitignore at master · github/gitignore

Python graph library

graph-tool: Efficent network analysis with python looks like a really good and modern graph theory library for python


  1. Taskwarrior - FAQ ↩︎

  2. git ignore vim temporary files - Stack Overflow ↩︎

Day 882

Docker mounting when developing, so as not to rebuild the image after each change

You Don’t Need to Rebuild Your Development Docker Image on Every Code Change · vsupalov.com

Pytorch memory leak when doing CPU inference

Got solved by using jemalloc instead of malloc. … No idea why and how that works.

Linux youtube client “red” / “utube”

keshavbhatt/red: Red - Privacy focused Youtube player and download manager for Linux, uses youtube-dl as backend. afaik it’s snap-only.

Unstable and crashes a lot though :(

Day 881

python glances

Glances · PyPI is a htop-like monitoring thingy.

Day 878

qutebrowser clear data for a specific website

Can be done through dev tools! Clear all site data, just cookies, or anything else. [^qbprivgithub ]

Learning git

Will be using the old and awesome Git - Book and a small test local repo.

2.2 Git Basics

Git file status

git status -s is short git status

Day 877

Docker DEBIAN_FRONTEND=noninteractive

Setting it in Dockerfiles is discouraged (even by the official Docker FAQ 1) because it’s mainly cosmetic & may create unwanted side effects.

For me, tzdata wanted input and waited for it:

[17:01:56][Step 1/3] debconf: falling back to frontend: Readline
[17:01:56][Step 1/3] Configuring tzdata
[17:01:56][Step 1/3] ------------------
[17:01:56][Step 1/3] 
[17:01:56][Step 1/3] Please select the geographic area in which you live. Subsequent configuration
[17:01:56][Step 1/3] questions will narrow this down by presenting a list of cities, representing
[17:01:56][Step 1/3] the time zones in which they are located.
[17:01:56][Step 1/3] 
[17:01:56][Step 1/3]   1. Africa      4. Australia  7. Atlantic  10. Pacific  13. Etc
[17:01:56][Step 1/3]   2. America     5. Arctic     8. Europe    11. SystemV
[17:01:56][Step 1/3]   3. Antarctica  6. Asia       9. Indian    12. US

Fixed this by adding this command specifically before the one requiring it:

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y

vaex - faster panda-like lib

TODO: Vaex: Pandas but 1000x faster - KDnuggets

Looks interesting. Why is it faster?

python subprocess run

subprocess.run() is the newer version of ..call(). Can run a string like this:

subprocess.run("echo  one two three", shell=True)

Qutebrowser throwaway email and password generatorr userscripts

Generate password, paste it into a textfield, and xclip the output:

#!/usr/bin/python3
import os
import string
import secrets
from subprocess import run
alphabet = string.ascii_letters + string.digits
password = ''.join(secrets.choice(alphabet) for i in range(8))

run(f"echo {password} | xclip -selection c", shell=True)
with open(os.environ['QUTE_FIFO'], 'w') as f:
    f.write(":insert-text {}".format(password))

Generate a throwaway email with email based on domain (so if I were to run it on google.com, it’d generate google@wildcard.mydomain.net:

#!/usr/bin/python3
import os
import tldextract
import argparse
import sys

argument_parser = argparse.ArgumentParser()
argument_parser.add_argument('--subdomain', '-s', default='t',
                             help='subdomain ("t" would do "@t.email_host.net")')
argument_parser.add_argument('--email_host', '-d', default='email_host.net',
                             help='main domain where you\'ll get the emails')
argument_parser.add_argument('--username', '-u', default=None,
                             help='the name used for email username (name@...)')
def main(args):
    my_domain = args.email_host
    subdomain = args.subdomain
    if args.username is not None:
        username = args.username
    else:
        url = os.environ['QUTE_URL']
        extract_result = tldextract.extract(url)
        username = extract_result.domain

    address = f"{username}@{subdomain}.{my_domain}"

    with open(os.environ['QUTE_FIFO'], 'w') as f:
        f.write(":insert-text {}".format(address))

if __name__ == '__main__':
    arguments = argument_parser.parse_args()
    sys.exit(main(arguments))

Use-case for both - quick easy registration in pointless places.


  1. Docker frequently asked questions (FAQ) | Docker Documentation ↩︎

Day 874

i3status VPN

My older approach was to use this:

run_watch VPN {
        pidfile = "/etc/openvpn/mv.pid"
}

And start openvpn in a way that it writes that specific pid file.

i3: i3status(1)’s documentation points at this:

path_exists VPN {
        # path exists when a VPN tunnel launched by nmcli/nm-applet is active
        path = "/proc/sys/net/ipv4/conf/tun0"
}

On my computer it was tap0 instead of tun0. But it works!

stow symlinks/targets

My ~/.dotfiles is a symlink to another place. stow follows it, and uses as target the parent directory of the directory the symlink points to, not ~/!

Explicitly setting a target directory is stow -t ~/ thing-to-stow (interestingly, stow -t ../ also uses the parent directory relative to the symlink target of the current one).

First I did the logical thing:

alias st='stow -t ~/'

Then, after reading the manual1, created a ~/.stowrc:

--target=~/

Works now :)

Wallabag tagging rules

Wallabag supports tagging rules based on parameters, such as domain names or reading time. Nice!

qutebrowser wallabag bookmarklet

Added ww as binding to the bookmarklet.

Fiamma qutebrowser-specific vimrc

I finally moved Fiamma (my link wiki) to a the new server! Which reminded me about the bindings I wrote to automatically format the input for the links I add there.

For example, on Ron Burk: Commas Depend on Linebreaks - Fiamma, I edited the pre-filled things to look like this:

http://ronburk.blogspot.de/2009/09/commas-depend-on-linebreaks.html
Ron Burk: Commas Depend on Linebreaks
6
5

language, linguistics, internet, style, etiquette, mildly interesting

Language
Style

Then a vim snippet from hell transformed it to

{{B|
http://ronburk.blogspot.de/2009/09/commas-depend-on-linebreaks.html
|Ron Burk: Commas Depend on Linebreaks
|6
|5
}}
{{#set:
k=language, linguistics, internet, style, etiquette, mildly interesting
|+sep=, }}

[[Category: Language]]
[[Category: Style]]

Though they were in latin-1 encoding, the .vimrc got converted to utf8, and it all got lost.

Now I have a solution. ~/.config/qutebrowser/.qb-vimrc is:

source ~/.vimrc

" let @H = 'gg<80>ýc<80>ýbi<80>ýc<80>ýb{{B|^[^[^[j0i|^[^[^[ji|j<80>kb^[^[^[ji|^[^[^[o}};q' " For the 5 lines
" let @L = 'ji{{$<80>kb%<80>kb#set:\^Mk=<80>kD^[o|+sep=,}}^[' " For the tags
" let @C = 'i[[C;tj<80>kb<80>kb<80>kbategory: ^[^[^[A]];q' " For each individual category
" let @F = 'jjVG:norm! @C\^M' "Apply that to all lines till the end
" let @d = '@H@L@F'
" let @q = '^[A^[bbbbbbi|<80>ü^B<80>kb^[:%s/=/{{=}}/ge^M'

" Summed up:
let @C = 'i[[C;tj<80>kb<80>kb<80>kbategory: ^[^[^[A]];q' " For each individual category
"let @H = '^[A^[bbbbbbi|<80>ü^B<80>kb^[:%s/=/{{=}}/ge^Mgg<80>ýc<80>ýbi<80>ýc<80>ýb{{B|^[^[^[j0i|^[^[^[ji|j<80>kb^[^[^[ji|^[^[^[o}};qji{{$<80>kb%<80>kb#set:^Mk=<80>kD^[o|+sep=,}}^[jjVG:norm! @C^M:x^M'
let @H = '^[A^[bbbbbbi|<80>ü^B<80>kb^[:%s/=/{{=}}/ge^Mgg<80>ýc<80>ýbi<80>ýc<80>ýb{{B|^[^[^[j0i|^[^[^[ji|j<80>kb^[^[^[ji|^[^[^[o}};qji{{$<80>kb%<80>kb#set:^Mk=<80>kD^[o|+sep=,}}^[jjVG:norm! @C^M' " Without closing at the end
" let @d = '@H@L@F'

" Start in insert mode
startinsert

And in qutebrowser config, I set the editor to:

c.editor.command = ['kitty', 'vim', '-u', str(config.configdir / '.qb-vimrc'), '+{line}', '{file}']

This way, standard-vim uses the standard fancy utf8 config file, but qutebrowser uses a separate one that overwrites the needed lines with the latin-1 macros. vim +10 filename means open it and put the cursor on line 10, idea comes from Reddit[^ideared

(Macros are really hard to read. How can I use something like python next time for this?)

Also - them being defined in the ~/.vimrc seems to have broken the newer ones, had to comment them out. Does vim not like redefined macros?

Updated my yank-for-markdown yank.py userscript to remove the anchor text ("…#!~:text=Text on the page to scroll to"), so I can paste it without it messing up the markdown formatting:

#!/usr/bin/python3
import os

title = os.environ['QUTE_TITLE']
title = title.replace("|", "\\|")

url = os.environ['QUTE_URL']
url = url.split("#:~:")[0]

command = "yank inline \"[{}]({})\"".format(title, url)

with open(os.environ['QUTE_FIFO'], 'w') as f:
    f.write(command)

Better Fiamma page creation with preloading

Rewrote the whole mechanism, now there’s one template that gets pre-filled by URI. First the qb userscript gets the data, writes them to a file; then opens this file in vim. When closed, it calls the new template passing the entire content of the file as first parameter.

Better because much simpler and less steps needed.

Random / quotes

[23:07:35] i mean, i have important work to do. dealing with an IRC network is not really something i want to be doing this decade outside of fucking around for fun with IRCX [23:07:51] i have code running on two planets 2


  1. Resource Files (Stow) ↩︎

  2. A gem from one of the linked chatlogs (Ariadne is security chair for Alpine linu… | Hacker News ↩︎

Day 873

Qutebrowser crashing fix

I think I have this time - removing state got it to start without reinstalling/changing anything.

Using screen in places that don’t support screen

Figured out myself and kinda proud of this one. If server1 doesn’t have screen, you can ssh to it from inside screen of a server2 that does have screen! As long as the SSH connection is there it’ll work.

json dump of np.float32

When doing jsons.dumps(thing) where thing has np.float32s inside it, you get the error:

TypeError: Object of type 'float32' is not JSON serializable

This is fixed by:

  • doing json.dumps(str(thing)) (though will return it as string, may or may not be what we want)
  • Converting the np.float32s to standard python float before adding them to the object

Mosquito / MQTT / openHAB

  • mosquito is an ubuntu implementation of the mqtt protocol, which is “subscribe to a broker for messages of type X and you’ll get them” - seems to be a standard like REST.
  • OpenHAB is a self-hosted thing that nicely manages such endpoints

(from V.H’s presentation about “Как подключить вайфай к чайнику для чайников”)

NLTK preprocessing for German

German tutorial about preprocessing German with NLTK: Preprocessing

zsh add binding to edit in vim

Added a zsh binding that in vi command mode launches edit-command-line to edit the current line in vim proper:

bindkey -M vicmd v edit-command-line

Doesn’t conflict with zsh-vim-mode-plugin. It’s nice how they all build upon the existing zsh infrastructure and I can keep adding my own bindings using the same mechanisms.

Day 869

BERT pytorch HF/HuggingFace NER Tensorboard

It puts the tensorboard files in ./runs of the directory I’m running the script from, not the output directory!

kitty hints

If there are a lot, the closest one to the cursor is marked , and can be selected by pressing <Enter>

qutebrowser browsing history

Started with a new profile, and realized how much I relied on it. Apparently suggestiosn based on browsing history is integral to my productivity

Vim sort lines

Highlight the wanted lines, then :sort!

This might be a place to look for similar vim commands: Vim documentation: change

Day 867

Bash split textfile by percentage

Split: how to split into different percentages? - Unix & Linux Stack Exchange:

split -l $[ $(wc -l filename|cut -d" " -f1) * 70 / 100 ] filename 

This creates files called xaa and xab and works fine for my purposes.

POSIX standard for shells/utilities

Introduction - TIL that head doesn’t really follow them

Day 864

zsh bracketed paste (don’t run command in terminal when pasting)

Stop terminal auto executing when pasting a command - Ask Ubuntu:

  • If you copy a newline symbol at the end of whatever you are copying, it gets executed as expected
  • bracketed paste (enabled by default on zsh) disables this behaviour

Had unset zle_bracketed_paste in zsh config, likely needed for athame that I don’t use. Removed it, works now.

To enable in bash,

echo "set enable-bracketed-paste" >> .inputrc

I should make an eventual list of dotfiles I use for all remote servers, this will go there 100%.

Docker COPY copies contents, not directory

Docker COPY copies contents, not directory \ Docker COPY copies contents, not directory \ Docker COPY copies contents, not directory \ Docker COPY copies contents, not directory \

kitty hint for IPs + python non-capturing (unnamed?) groups

Added these to kitty config! One for IPs, second IPs+ports:

map kitty_mod+n>i kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3} --program @
map kitty_mod+n>p kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3}:[0-9]+ --program @

Glad I can still read and understand regexes. The above highlight more than needed, but seems to be kitty’s problem.

In python, a group without ?: is a non-capturing group in python (= not returned in .groups()). In kitty (that uses python syntax), only what’s inside the first capturing group is copied; making it non-capturing makes it copy the entire regex. 1

I added another kitty hint to copy CLI commands currently being typed:

# CLI Commands
map kitty_mod+n>c kitten hints --type regex --regex "\$(.+)\s*$" --program @

My regex is trivial, the capturing group gets the command without the leading $ and avoids all trailing whitespaces.

Docker run detached mode

The magic -dp 8000:8000 command I’ve been using is actually -d -p, with -p being what I want and -d turning on detached mode. Without it, I see the logs directly and can easily <Ctrl-c> it away.

Also, docker ps shows ports as part of the output.

Setting timezone

Let this be the final one, with all configs correct now:

timedatectl set-timezone Europe/XXX

Quotes

In the Buddhist interpretation of it, “BE WHERE YOU ARE”.


  1. Hints — kitty 0.20.3 documentation ↩︎

Day 863

Remapping a Thinkpad T580 Fn key to Ctrl

The location of the Fn key on the laptop keyboard is absolutely idiotic and I hate it. Fn keys are usually handled by the hardware and ergo unusable. Now that I have to use the keyboard more, thought I have nothing to lose and tried xev and oh what a wonderful world it gets read as XF86WakeUp! Therefore it can be remapped to something more sensible. … like the Ctrl key it should be.

Easiest way for me was adding this to autostart:

xcape -e 'XF86WakeUp=Control_L' -d &

No side effects of the other xcape command xcape -e 'Control_L=Escape' -t 100, it seems to be considered a different Control_L key and clicking it fast doesn’t produce Escape.

Day 862

Disable touchpad

xinput set-prop 13 340 1, where 13 comes from xinput -list

Dockefile RUN a lot of commands

It’s possible to do this instead of prefixing each command with RUN:

RUN apt-get update && \
    # install base packages
    apt-get install -y -qq apt-utils aptitude wget curl zip unzip sudo kmod git && \
    /usr/bin/python3 -m pip install --upgrade pip && \

Day 861

kitty hints

Changed the hint I most often use to a better binding:

# Copy url
# map kitty_mod+n>c kitten hints --type path --program @
map kitty_mod+g kitten hints --type path --program @

Timewarrior

  • w track 1728 tag1 automatically ends it `now``.
  • w continue just continues the last thing running by starting something identical starting “now” and continuing till stopped.

kitty kittens

kitty autocompletion

In zshrc:

autoload -Uz compinit
compinit
# Completion for kitty
kitty + complete setup zsh | source /dev/stdin

kitty scrollback pager

From Feature Request: Ability to select text with the keyboard (vim-like) · Issue #719 · kovidgoyal/kitty · GitHub:

scrollback_pager vim - -c 'w! /tmp/kitty_scrollback' -c 'term ++curwin cat /tmp/kitty_scrollback'

Vim 8.0 works. Nice colorful etc.

zsh vim mode timeout

Zsh Vi Mode:

Adding this allows to register the <Esc> key in 0.1 sec, not default 0.4.

export KEYTIMEOUT=1

A good documented vimrc

A Good Vimrc - TODO

I also love his design!

zsh vim mode with objects!

GitHub - softmoth/zsh-vim-mode: Friendly bindings for ZSH’s vi mode

Out of all the various vim plugins, this is the only one I found that allows to meaningfully work with objects, like ci' etc. Also the mode indicator works very reliably.

Doesn’t conflict with zsh-evil-registers.

English / random

  • “expect and require”

Day 860

Qutebrowser crashing - again

Ubuntu 18.04, qutebrowser etc, as usual. What helped was creating the environment with these options:

python3 scripts/mkvenv.py --pyqt-version 5.14

jq | less zsh alias

Should’ve done this a long time ago:

lq() {
    jq . "$1" -C | less
}

kitty terminal copy url

From config; I should use them more.

# Select a filename and copy it 
map kitty_mod+p>c kitten hints --type path --program @
#: Select a path/filename and open it with the default open program.
map kitty_mod+p>o kitten hints --type line --program -

update-alternatives & installing another gcc

Nicely described: How to switch between multiple GCC and G++ compiler versions on Ubuntu 20.04 LTS Focal Fossa - LinuxConfig.org

# install stuff
$ sudo apt -y install gcc-7 g++-7 gcc-8 g++-8 gcc-9 g++-9
# Add it to update-alternatives
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 7
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 7
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 8
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 8
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 9
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-9 9

# choose the default one
$ sudo update-alternatives --config gcc
There are 3 choices for the alternative gcc (providing /usr/bin/gcc).

  Selection    Path            Priority   Status
------------------------------------------------------------
  0            /usr/bin/gcc-9   9         auto mode
  1            /usr/bin/gcc-7   7         manual mode
* 2            /usr/bin/gcc-8   8         manual mode
  3            /usr/bin/gcc-9   9         manual mode
Press  to keep the current choice[*], or type selection number:

From the docs: --install link name path priority

Python pip

Editable installations (pip install -e .) are a thing. TODO - learn more about them.

Qutebrowser config - adding bindings for tabs 20-30

Given that the standard ones are not enough for me, and even my additional ones for 10-20 are not enough, added a third level:

config.bind('1', 'tab-focus 1')
config.bind('2', 'tab-focus 2')
config.bind('3', 'tab-focus 3')
config.bind('4', 'tab-focus 4')
config.bind('5', 'tab-focus 5')
config.bind('6', 'tab-focus 6')
config.bind('7', 'tab-focus 7')
config.bind('8', 'tab-focus 8')
config.bind('9', 'tab-focus 9')
config.bind('0', 'tab-focus 10')
config.bind('<Alt-1>', 'tab-focus 11')
config.bind('<Alt-2>', 'tab-focus 12')
config.bind('<Alt-3>', 'tab-focus 13')
config.bind('<Alt-4>', 'tab-focus 14')
config.bind('<Alt-5>', 'tab-focus 15')
config.bind('<Alt-6>', 'tab-focus 16')
config.bind('<Alt-7>', 'tab-focus 17')
config.bind('<Alt-8>', 'tab-focus 18')
config.bind('<Alt-9>', 'tab-focus 19')
config.bind('<Alt-0>', 'tab-focus 20')
config.bind('<Alt-Ctrl-1>', 'tab-focus 21')
config.bind('<Alt-Ctrl-2>', 'tab-focus 22')
config.bind('<Alt-Ctrl-3>', 'tab-focus 23')
config.bind('<Alt-Ctrl-4>', 'tab-focus 24')
config.bind('<Alt-Ctrl-5>', 'tab-focus 25')
config.bind('<Alt-Ctrl-6>', 'tab-focus 26')
config.bind('<Alt-Ctrl-7>', 'tab-focus 27')
config.bind('<Alt-Ctrl-8>', 'tab-focus 28')
config.bind('<Alt-Ctrl-9>', 'tab-focus 29')
config.bind('<Alt-Ctrl-0>', 'tab-focus -1')

EDIT: Actually, to think of it, in for a penny, in for a pound!

for i in range(30, 60):
    config.bind(','+str(i), 'tab-focus '+str(i))

Takes about 9 seconds to :config-source everything, but then works like a charm! And doesn’t seem to make anything else slower (strangely, even startup is as usual).

pycharm can parse markdown!

Opened a README.md, and see it being rendered nicely to the left. I can also edit it directly. Wow.

Website with references / cheat sheets for a lot of CLI programs

sed Cheat Sheet - very down-to-earth, “praxisnah”, I like it. Except for the idiotic scrolling override animations

jq basics - again

jq Cheat Sheet

  • I should use ' for the filter, " for any string elements inside it

  • select

    • Get full record if it matches something
    • jq '.results[] | select(.name == "John") | {age}' # Get age for 'John'
  • Value VS key-value

    • jq '.something' gets the content of fields something removing the key
    • jq '. | {something}' gets key-value of something
    • Sample:
$ jq '. | select(.tokens[0]=="Tel") | .tokens[]' mvs.json
"Tel"
":"
$ jq '. | select(.tokens[0]=="Tel") | .tokens' mvs.json
[
  "Tel",
  ":"
]
$ jq '. | select(.tokens[0]=="Tel") | {tokens}' mvs.json
{
  "tokens": [
    "Tel",
    ":"
  ]
}
  • |keys to extract keys only

jq Cheet Sheet · GitHub also nice TIl that you don’t need jq '. | keys', jq 'keys' etc is enough.

  • `‘del(.tokens)’ to delete a key
  • Indexing works like in Python, say jq '.[-2:]'
  • 'sort_by(.foo)'

I think now I’m ready for the holy of holies: jq 1.4 Manual

  • {user, title: .titles[]} will return an array of {user, title} for each value inside .titles[]!
  • Putting ()s around an expression means it’ll be evaluated. {(.user): .titles} will use the value of the key user!
$  jq '. | {(.id): .id}' mvs.json
{
  "7574": "7574"
}
  • Putting values inside strings with \(foo)
$ echo "[1,2,3]" | jq '"A string \(.)"'
"A string [1,2,3]"

It’s basically synonymous to python3’s f"My f-{string}"

  • '.a=23' will produce an output with .a being set to 23. Will be created if not there.
    • No “change” is being done, the actual value is the same; .a in the same filter after a comma will still return the old value.
  • |= will “update” the value by running its previous value through the expression:
$ echo '{"one": 23,"two":2}' | jq '.one|=(. | tostring)'
{
  "one": "23",
  "two": 2
}
  • slurp mode - instead of returning objects, return a list of objects! For more ‘correct’ json.

Python JSON parser + jq compact mode

It didn’t read the jq-generated multi-line output without commas between items, but jq compact mode does one record (without comma and not as part of an array) per line, and this gets parsed correctly!

JQ compact mode is jq -c '.' sth.json

Before:

{
  "id": "7575",
  "ner_tags": [
    "6",
    "6"
  ],
  "tokens": [
    "Tel",
    ":"
  ]
}

After:

{"id":"7575","ner_tags":["6","6"],"tokens":["Tel",":"]}

Linux - creating a directory accessible to multiple users via a group

How to Create a Shared Directory for All Users in Linux

# Create the group
$sudo groupadd project 
# Add user to this group
$sudo usermod -a -G project theuser
# Change the group of the directory
$ sudo chgrp -R project /var/www/reports/
# Turn on the `setGID` bit, so newly created subfiles inherit the same group as the directory
# And rwxrwx-rx
$ sudo chmod -R 2775 /var/www/reports/

Day 856

Presenting stuff

“Which story do you want to tell?” (Heard at work, from R)

Git get commit message from file

git commit -F filename allows to use a pre-written commit message from a textfile.

Day 855

i3 scratchpads magic!

You can ‘mark’ windows1, a la vim, and then use that as filter - no window classes etc needed - for example, for scratchpads!2

So now I have two scratchpads in i3 config:

bindsym $ms+Shift+plus mark "scratch2", move scratchpad
bindsym $ms+plus [con_mark="scratch2"]  scratchpad show

bindsym $ms+Shift+minus mark "scratch", move scratchpad
bindsym $ms+minus [con_mark="scratch"]  scratchpad show

The second one originally was meant to be for Ding, but it’s really nice to have it flexible.


  1. i3: i3 User’s Guide ↩︎

  2. Marks + Scratchpad = Awesome : i3wm ↩︎

Day 854

English

Reading “German: An Essential Grammar” by Donaldson found this bit: 1

English has a rule that if the time of an event that
occurred in the past is mentioned, then the imperfect must be used, but if
the time is omitted, the perfect is required, e.g. \

  • He returned from Hamburg yesterday.
  • He has returned from Hamburg.
  • He has returned from Hamburg yesterday. (not grammatical)

TIL.

zsh detach and disown

zsh-specific - to detach & disown a process, there’s &!: 2

dolphin &!

German / Deutsch

Long question and answer about fahren zu/nach/in/…: Richtungen und Ziele

German FSI language courses

The Yojik Website has the FSI courses FSI Languages Courses and the website as I remember it.

Taskwarrior

Changed ~/.taskrc to show any active tasks regardless of anything else in my sprint view:

s () {task s \(project:w or \(sprint:$SPRINT \(+A or +O\)\) or +ACTIVE\) "$*"}

Turn off screen/monitor with xset

Standard lock command leaves both monitors on.

Reddit3 mentioned two commands:

xset s activate
xset dpms force off

The second one worked for me!

Now I have shiny new screen lock (and suspend too, while we are at it) keybinding in i3 config!

bindsym $ms+n exec gnome-screensaver-command -l && xset dpms force off
bindsym $ms+Shift+n exec i3lock -i ~/s/black_lock.png -t -p win -e && systemctl suspend -i

  1. p. 118 ↩︎

  2. bash - Exit zsh, but leave running jobs open? - Stack Overflow ↩︎

  3. Turn off screen after a moment if locked : i3wm ↩︎

Day 853

Nvidia Docker images

Nvidia has a repo of all docker images it creates, one of them: Torch | NVIDIA NGC

German

“Das finde ich zielführender als…” - heard at work

Docker - automatically assign a free port

docker run --name frontend -p 0:80 frontend:latest1

Port 0 gets passed to the kernel that assigns any free port.

To see which one, docker port somecontainer.

Docker run container on specific GPU

docker run --gpus device=3 -e NVIDIA_VISIBLE_DEVICES=0 -e CUDA_VISIBLE_DEVICES=0 myservice

Where the device=3 is the GPU id on the host that we want to use.


  1. docker - Bash command to return a free port - Stack Overflow ↩︎

Day 850

grep ignore case

lspci | grep -i "nvidia"

-i == ‘ignore case’ is actually something that I can remember.

Docker (stop) autostart of container

Docker will autostart any container with a RestartPolicy of ‘always’ when the docker service initially starts. 1

I can set/unset it in kitematic, or through terminal:

docker update --restart=no my-container

apt-get purge remove –autoremove etc

Quoting SO: 2

    apt purge --auto-remove <packagename>

purges packagename and any packages which are rendered unnecessary by its removal, as well as any other packages which aren’t necessary.

    apt autoremove --purge

purges any packages which aren’t necessary (marked as “automatically installed” and with no dependent packages).

The first form is what you’d use when manipulating individual packages; the latter is a clean-up operation across all packages.

Ways to clean up with apt-get - tutorial

This seems nice, TODO: Cleaning up with apt-get | Network World

Backing up LVM disk encryption keys

LVM - Debian Wiki is nice and readable. I used this command to backup the headers:

 sudo cryptsetup luksHeaderBackup /dev/nvmeXXXXX   --header-backup-file headerBackupFile

… and put it somewhere not on the drive I’ll be recovering if it all goes wrong.

Setting up Tensorflow and CUDA with an eGPU

Aaaand the saga continues!

…since the GPU is an eGPU, apparently I do need to do the harder way: Accelerating Machine Learning on a Linux Laptop with an External GPU | NVIDIA Developer Blog

Getting the eGPU detected

It is, I can see it:

(17:42:42/10815)~/$ lspci | grep -i VGA
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 620 (rev 07)
0c:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)

but if it wasn’t, I’d authorize it and check with boltctl list:

(17:43:13/10817)~/$ boltctl list
[...]
 ● GIGABYTE GV-N1070IXEB-8GD
   ├─ type:          peripheral
   ├─ name:          GV-N1070IXEB-8GD
   ├─ vendor:        GIGABYTE
   ├─ uuid:          # redacted
   ├─ status:        authorized
   │  ├─ domain:     domain0
   │  └─ authflags:  none
   ├─ authorized:    Do 29 Apr 2021 07:57:37 UTC
   ├─ connected:     Do 29 Apr 2021 07:57:37 UTC
   └─ stored:        no

How to setup an eGPU on Ubuntu for TensorFlow describes other things that can go wrong:

I had to disable the following, otherwise my eGPU was not detected:

  • Secure Boot
  • Thunderbolt Security Level

From this point on, I follow Nvidia’s tutorial 3 unless stated otherwise.

Purging, cleaning up old broken install attempts, updating and upgrading

Using quotes means the * doesn’t have to be escaped.

sudo apt-get purge "nvidia*"

This is a fuller example: 4

sudo rm /etc/apt/sources.list.d/cuda*
sudo apt remove --autoremove nvidia-cuda-toolkit
sudo apt remove --autoremove nvidia-*

Found and manually removed /etc/apt/sources.list.d/graphics-drivers-ubuntu-ppa-bionic.list, leaving the .save file in place.

As per nvidia’s guide,

sudo apt-get update
sudo apt-get dist-upgrade

To be safe, rebooted.

Downloading the correct drivers

The existing driver is most likely Nouveau, an open-source driver for NVIDIA GPUs. Because Nouveau doesn’t support eGPU setups, install the NVIDIA CUDA and NVIDIA drivers instead. You must also stop the kernel from loading Nouveau. 3

okay!

Change of plan - what is NVIDIA data-science-stack?

Found this: NVIDIA/data-science-stack: NVIDIA Data Science stack tools Read about it here: Ubuntu for machine learning with NVIDIA RAPIDS in 10 min | Ubuntu

Official by nvidia, and seems to do automatically what’s needed for supported systems. Let’s run a script from the internet that installs drivers, loads kernel modules etc.

Source is available, yay for open source: data-science-stack/data-science-stack at master · NVIDIA/data-science-stack

Ran ./data-science-stack setup-system - uses sudo, didn’t ask for root or anything.o

Seems to have installed nvidia driver version 460. Asked to reboot at the end.

Rebooted.

(18:40:30/10909)~/$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

okay. Same results I had. Confirms that my prev. steps weren’t wronger than the script.

(18:41:49/10910)~/$ sudo apt list --installed | grep "\(cuda\|nvidia\)"

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnccl2/unknown,now 2.9.6-1+cuda11.3 amd64 [installed]
libnvidia-cfg1-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
libnvidia-common-460/unknown,now 460.73.01-0ubuntu1 all [installed,automatic]
libnvidia-compute-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
libnvidia-container-tools/bionic,now 1.4.0-1 amd64 [installed,automatic]
libnvidia-container1/bionic,now 1.4.0-1 amd64 [installed,automatic]
libnvidia-decode-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
libnvidia-encode-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
libnvidia-extra-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
libnvidia-fbc1-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
libnvidia-gl-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
libnvidia-ifr1-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
nvidia-compute-utils-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
nvidia-container-runtime/bionic,now 3.5.0-1 amd64 [installed,automatic]
nvidia-container-toolkit/bionic,now 1.5.0-1 amd64 [installed,automatic]
nvidia-dkms-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
nvidia-docker2/bionic,now 2.6.0-1 all [installed]
nvidia-driver-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed]
nvidia-kernel-common-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
nvidia-kernel-source-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
nvidia-prime/bionic-updates,bionic-updates,now 0.8.16~0.18.04.1 all [installed,automatic]
nvidia-settings/unknown,unknown,now 465.19.01-0ubuntu1 amd64 [installed,automatic]
nvidia-utils-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]
xserver-xorg-video-nvidia-460/unknown,now 460.73.01-0ubuntu1 amd64 [installed,automatic]

Also, as usual,

(18:48:34/10919)~/$ lsmod | grep nvi
(18:48:37/10920)~/$

lspci -k shows the kernel modules:

0c:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd GP104 [GeForce GTX 1070]
        Kernel modules: nvidiafb, nouveau

This output implies no nvidia driver is installed on my system5. …though it is.

$ nvidia-settings --version
nvidia-settings:  version 465.19.01

software-properties-gtk tells me I’m using the proprietary nvidia-driver-460, not 465

In any case, can’t blacklist nouveau as still there are no ubuntu kernel modules.

BUT!

(19:04:04/10946)~/$ dkms status
nvidia, 460.73.01: added

Also, inxi -Fxxxrz (found somewhere on the internet):

Graphics:  Card-1: Intel UHD Graphics 620 bus-ID: 00:02.0 chip-ID: 8086:5917
           Card-2: NVIDIA GP104 [GeForce GTX 1070] bus-ID: 0c:00.0 chip-ID: 10de:1b81
           Display Server: x11 (X.Org 1.19.6 ) drivers: modesetting,nvidia (unloaded: fbdev,vesa,nouveau)

It it sees them as there and loaded? Does dkms somehow bypass lsmod etc?

sudo dkms autoinstall should autoinstall all added drivers, …let’s hope for the best I guess.

(19:11:47/10958)~/$ sudo dkms autoinstall

Kernel preparation unnecessary for this kernel.  Skipping...
applying patch disable_fstack-clash-protection_fcf-protection.patch...patching file Kbuild
Hunk #1 succeeded at 85 (offset 14 lines).


Building module:
cleaning build area...
unset ARCH; [ ! -h /usr/bin/cc ] && export CC=/usr/bin/gcc; env NV_VERBOSE=1 'make' -j8 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=5.4.0-72-generic IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1 SYSSRC=/lib/modules/5.4.0-72-generic/build LD=/usr/bin/ld.bfd modules......(bad exit status: 2)
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/nvidia-dkms-460.0.crash'
Error! Bad return status for module build on kernel: 5.4.0-72-generic (x86_64)
Consult /var/lib/dkms/nvidia/460.73.01/build/make.log for more information.

The file is long, keys seems:

 scripts/Makefile.build:269: recipe for target '/var/lib/dkms/nvidia/460.73.01/build/nvidia/nv.o' failed
 make[2]: *** [/var/lib/dkms/nvidia/460.73.01/build/nvidia/nv.o] Error 1
 Makefile:1754: recipe for target '/var/lib/dkms/nvidia/460.73.01/build' failed
 make[1]: *** [/var/lib/dkms/nvidia/460.73.01/build] Error 2
 make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-72-generic'
 Makefile:80: recipe for target 'modules' failed
 make: *** [modules] Error 2
DKMSKernelVersion: 5.4.0-72-generic
Date: Fri Apr 30 18:30:45 2021
DuplicateSignature: dkms:nvidia-dkms-460:460.73.01-0ubuntu1:/var/lib/dkms/nvidia/460.73.01/build/conftest/functions.h:11:2: error: #error acpi_walk_namespace() conftest failed!
Package: nvidia-dkms-460 460.73.01-0ubuntu1
PackageVersion: 460.73.01-0ubuntu1
SourcePackage: nvidia-graphics-drivers-460
Title: nvidia-dkms-460 460.73.01-0ubuntu1: nvidia kernel module failed to build

Smells like a driver/kernel support isse?

First result when googling dkms nvidia 460 is this: Can’t get nvidia 460 module to build on Ubuntu 20.04 to support two A100s - GPU Unix Graphics / Linux - NVIDIA Developer Forums

Please check if the build symlink to the headers for dkms exists:

ls /lib/modules/$(uname -r)/build

Otherwise, create it

ln -s /usr/src/linux-headers-$(uname -r)  /lib/modules/$(uname -r)/build

Didn’t have it, created it, trying again, same error, deleted the previous log, full output is:

(19:19:54/10967)~/$ sudo dkms autoinstall

Kernel preparation unnecessary for this kernel.  Skipping...
applying patch disable_fstack-clash-protection_fcf-protection.patch...patching file Kbuild
Hunk #1 succeeded at 85 (offset 14 lines).


Building module:
cleaning build area...
unset ARCH; [ ! -h /usr/bin/cc ] && export CC=/usr/bin/gcc; env NV_VERBOSE=1 'make' -j8 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=5.4.0-72-generic IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1 SYSSRC=/lib/modules/5.4.0-72-generic/build LD=/usr/bin/ld.bfd modules.......(bad exit status: 2)
Error! Bad return status for module build on kernel: 5.4.0-72-generic (x86_64)
Consult /var/lib/dkms/nvidia/460.73.01/build/make.log for more information.

The file is full of what looks like syntax errors..?

This charming chinese website seems to imply gcc version is to blame: NVIDIA驱动出错:NVIDIA-SMI has failed because it couldn‘t communicate with the NVIDIA driver. Make sure t_sazass的博客-CSDN博客

(19:22:39/10974)~/$ cat /proc/version
Linux version 5.4.0-72-generic (buildd@lgw01-amd64-021) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #80~18.04.1-Ubuntu SMP Mon Apr 12 23:26:25 UTC 2021
sudo apt install gcc-8
sudo update-alternatives --config gcc
sudo update-alternatives --remove-all gcc
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 10
sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-8 10

Let’s retry dkms autoinstall:

(19:26:03/10981)~/$ sudo dkms autoinstall

Kernel preparation unnecessary for this kernel.  Skipping...
applying patch disable_fstack-clash-protection_fcf-protection.patch...patching file Kbuild
Hunk #1 succeeded at 85 (offset 14 lines).


Building module:
cleaning build area...
unset ARCH; [ ! -h /usr/bin/cc ] && export CC=/usr/bin/gcc; env NV_VERBOSE=1 'make' -j8 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=5.4.0-72-generic IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1 SYSSRC=/lib/modules/5.4.0-72-generic/build LD=/usr/bin/ld.bfd modules...............
Signing module:
 - /var/lib/dkms/nvidia/460.73.01/5.4.0-72-generic/x86_64/module/nvidia-modeset.ko
 - /var/lib/dkms/nvidia/460.73.01/5.4.0-72-generic/x86_64/module/nvidia.ko
 - /var/lib/dkms/nvidia/460.73.01/5.4.0-72-generic/x86_64/module/nvidia-uvm.ko
 - /var/lib/dkms/nvidia/460.73.01/5.4.0-72-generic/x86_64/module/nvidia-drm.ko
Secure Boot not enabled on this system.
cleaning build area...

DKMS: build completed.

nvidia.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-72-generic/updates/dkms/

nvidia-modeset.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-72-generic/updates/dkms/

nvidia-drm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-72-generic/updates/dkms/

nvidia-uvm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-72-generic/updates/dkms/

depmod...

DKMS: install completed.

WOW. WOOOOOW. WOOOOOOOOOOOOOOOOOOOOOO

Without even restarting, after the first command my screen flashed and changed resolution a bit, BUT THEN IT WORKED

(19:34:17/10983)~/$ nvidia-smi
No devices were found
(19:34:20/10984)~/$ nvidia-smi
Fri Apr 30 19:34:22 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    On   | 00000000:0C:00.0 Off |                  N/A |
|  0%   54C    P0    37W / 151W |      7MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

All these attempts failed because the nvidia module in dkms couldn’t install because syntax errors because old gcc compiler version.

What could I have done differently? Why at no point did I see errors about the kernel module failing to build, where should I have looked for them? And why syntax errors instead of something checking the used gcc version and loudly failing when there was a mismatch? Why is that chinese website the only place I found this fix?

(19:42:57/10995)~/$ lsmod | grep nvidia
nvidia_uvm           1015808  0
nvidia_drm             57344  1
nvidia_modeset       1228800  1 nvidia_drm
nvidia              34123776  17 nvidia_uvm,nvidia_modeset
drm_kms_helper        188416  2 nvidia_drm,i915
drm                   491520  15 drm_kms_helper,nvidia_drm,i915

Now let’s hope this survives a restart. And that it works when the eGPU is disconnected.

NVIDIA data-science-stack

Following the readme, ran both options in separate terminals:

./data-science-stack list
./data-science-stack build-container
./data-science-stack run-container

and

./data-science-stack list
./data-science-stack build-conda-env
./data-science-stack run-jupyter

The latter seems to be installing CUDA and friends on my computer - didn’t expect it, but I need them either way I think, I guess I’ll let the script handle everything since it started. It installed conda to ~/conda/, but again, not sure what I was expecting

Both running for 20+ minutes now

EDIT: ~/conda/ took 20gb filling up my drive, blocking everything, deleted it

The docker with jupyterlab - tensorflow can’t access the GPU, but pytorch can.

Carrying on with setting the eGPU up

The NVIDIA eGPU tutorial3 continues with offloading Xorg to the GPU - do I want this? Can I use the GPU just for training, and leave Xorg running on the internal one? I probably don’t

Restarting and testing

As I remember from the last time, X doesn’t start when the GPU is connected at boot but everything’s fine when it gets connected after starting X. When it’s connected, it seems the driver gets loaded and nvidia-smi etc works. That the system works without the eGPU attached is nice! Plug-and-play is nice too.

Installed pytorch in a virtualenv, for cuda 11.1, test snippet says cuda works!

import torch
x = torch.rand(5, 3)
print(x)

torch.cuda.is_available()

Tensorflow:

>>> import tensorflow as tf
2021-04-30 21:36:12.984883: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
>>> tf.debugging.set_log_device_placement(True)
>>> a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
2021-04-30 21:36:23.055614: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-30 21:36:23.058062: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-04-30 21:36:23.115366: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-30 21:36:23.116510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0c:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.721GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2021-04-30 21:36:23.116553: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-04-30 21:36:23.119974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-04-30 21:36:23.120034: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-04-30 21:36:23.121503: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-30 21:36:23.121842: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-30 21:36:23.125037: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-30 21:36:23.125803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-04-30 21:36:23.125980: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2021-04-30 21:36:23.125996: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

Which libcudnn?

Tensorflow’s tutorial (GPU support  |  TensorFlow) does this:

Install development and runtime libraries (~4GB)
sudo apt-get install --no-install-recommends \
    cuda-11-0 \
    libcudnn8=8.0.4.30-1+cuda11.0  \
    libcudnn8-dev=8.0.4.30-1+cuda11.0

What is the version for CUDA 11.2? cuDNN Archive | NVIDIA Developer has download links. The one for 11.2 is called “cudnn-11.2-linux-x64-v8.1.1.33.tgz”. I plug those versions in, they exist and install fine:

sudo apt-get install   libcudnn8=8.1.1.33-1+cuda11.2
sudo apt-get install   libcudnn8-dev=8.1.1.33-1+cuda11.2

And tensorflow now works!

2021-04-30 21:42:46.176942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7440 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:0c:00.0, compute capability: 6.1)

I can’t believe it but wow. It’s finished, it works, X didn’t die, plug-and-play works, no manual driver loading.

All in all, including all the failed attempts, took 5:30h of pure time, according to my time tracking.

The only wrinkle is that X doesn’t start when turning the computer on with the eGPU attached, but I can 100% live with that!

GPU benchmarking linux

How to Benchmark your GPU on Linux has a fun quote:

This tool is very old, very basic and only tests a small portion of today’s OpenGL capabilities. Back in the old days, it was used to determine if the proprietary driver was installed and running properly as open-source drivers were performing awfully enough to be perfectly noticeable during this test. Nowadays, you won’t notice any difference between the two

qutebrowser open a private window

Added this to config.py:

config.bind('<Alt-P>', 'set-cmd-text -s :open -p ')

Managing dotfiles with machine-specific configuration

Qutebrowser import other config files

Seen in someone’s config.py on gitlab6:

for f in glob.glob(str(config.configdir / 'conf.d/*.py')):
    config.source(str(os.path.relpath(f, start=config.configdir)))

Random i3 configs

Nice examples: i3_config/settings.d at master · kiddico/i3_config · GitHub

i3 doesn’t have any kind of include directive in the config files, sadly. i3 - Source/import file from i3wm config - Stack Overflow is one option:

bindsym $mod+Shift+c exec "cat ~/.config/i3/colors ~/.config/i3/base > ~/.config/i3/config && i3-msg reload"

A keybinding to overwrite the config file and restart i3 with a command.

To read - life hacking

This looks very interesting, I shouldn’t forget to go through this: Life Hacking His blog with personal examples: Alex Vermeer — Life-Hacking. Climbing. Striving for awesome. Coffee. — Page 2

A non-pdf description of Life Areas with questions and metrics for each.

(He’s the same guy who created the awesome How to Get Motivated: A Guide for Defeating Procrastination poster!)

And let’s remember the classic: Evidence-based advice on how to be successful in any job - 80,000 Hours

Detach process completely from terminal

Two options I like:7

  • nohup cmd &
  • cmd & disown

I feel one of these will become part of many aliases of mine.

And short bash function from the same place:

function dos() {
    # run_disowned and silenced

    run_disowned "$@" 1>/dev/null 2>/dev/null
}

  1. Docker – Prevent Container Autostart | eknori.de↩︎

  2. debian - What’s the right way to purge recursively with apt? - Unix & Linux Stack Exchange ↩︎

  3. Accelerating Machine Learning on a Linux Laptop with an External GPU | NVIDIA Developer Blog ↩︎

  4. CUDA 10.1 installation on Ubuntu 18.04 LTS | Medium ↩︎

  5. Install the Latest Nvidia Linux Driver - LinuxConfig.org ↩︎

  6. ~pvsr/dotfiles: qutebrowser/.config/qutebrowser/config.py - sourcehut git ↩︎

  7. linux - How do I detach a process from Terminal, entirely? - Super User ↩︎

Day 849

PEP8

To read: PEP 8 – Style Guide for Python Code | Python.org

English / random

  • “If you feel a misalignment with …”
  • Ticketize (verb)

Jira ticket search and filtering

I should learn about the search syntax for jira tickets:

assignee = currentuser() and statusCategory != Done ORDER BY updated DESC

Day 848

Installing CUDA and pytorch and tensorflow

Following this: CUDA 10.1 installation on Ubuntu 18.04 LTS | Medium nope, errors

In the same github discussion about installing CUDA on ubuntu that I’ve been to twice this bit is mentioned: 1

The very very important thing is that never install “nvidia-driver-***” driver by yourself.

Required nvidia drivers are installed while doing sudo apt install -y cuda=10.0.130-1

Zsh wildcards and apt-get remove

sudo apt remove --autoremove nvidia-* doesn’t work as-is in zsh! * gets interpreted as files in current directory. Explains my CUDA issues, everything seemed to work till I ran the above in a directory containing files with matching names that got helpfully shown.

sudo apt remove --autoremove nvidia-\* is the answer.

(or 'nvidia-*')

Not the first time this bites me, at least the third, and all of them in the context of doing CUDA stuff.

German

“Es funktioniert fabelhaft” - heard at work

Purging packages

apt --fix-broken install didn’t help as advertised, but removing all the broken packages together with sudo dpkg -P cuda-libraries-10-0 libnvidia-common-390 helped! After this removing/cleaning up everything else worked. A lot of this mentioned changes to initramfs, I really hope I’ll be able to boot up next time :(

Also - if 90% of the tutorials about how to install $thing start with “Remove any traces of installs of $thing you have” it’s a nice sign that something’s shady.

Docker logs

docker logs 09348209840239

i3 skype floating window fix

Skype fix : i3wm:

Option 1: hide the floating window:

for_window [title="^Skype$" floating] move scratchpad

Option 2:

Clever idea. Although, are you talking about the little window that can be disabled in Skype’s “Settings > Calling > Show call window when Skype is in the background”?

Slack show all messages in all channels

In search, before:Tomorrow is a nice catch-all filter

Pytorch installs its own CUDA!

Your system installations of CUDA and cudnn won’t be used, if you install PyTorch binaries with these libs. E.g. conda install pytorch torchvision cudatoolkit=10.1 -c pytorch will install CUDA 10.1 and cudnn in your current conda environment. 2

Tensorflow CUDA Docker doesn’t need CUDA on host machine, only the nvidia drivers

Nvidia drivers are needed on host machine, but not CUDA! 3

Random / UX / Design?

On TF’s official CUDA install page4, the bash listings (that are usually copypasted) contain the standard $ at the beginning, it’s visible, but not copypastable!

Installing CUDA 11.0 using official Tensorflow tutorial

So, hopefully the last time today, as the previous couple of times I end up in the official TF tutorial4 about installing CUDA. Armed with the knowledge that:

  • pytorch installs its own CUDA and doesn’t care, as long as GPU drivers are there
  • Docker installs its own CUDA and doesn’t care, as long as GPU drivers are on the host machine
  • Installing nvidia drivers should not be manual, it has to be done by the cuda packages

Snippet:

# Add NVIDIA package repositories
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
sudo apt-get update

wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
sudo apt-get update

wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/libnvinfer7_7.1.3-1+cuda11.0_amd64.deb
sudo apt install ./libnvinfer7_7.1.3-1+cuda11.0_amd64.deb
sudo apt-get update

# Install development and runtime libraries (~4GB)
sudo apt-get install --no-install-recommends \
    cuda-11-0 \
    libcudnn8=8.0.4.30-1+cuda11.0  \
    libcudnn8-dev=8.0.4.30-1+cuda11.0

# Reboot. Check that GPUs are visible using the command: nvidia-smi

# Install TensorRT. Requires that libcudnn8 is installed above.
sudo apt-get install -y --no-install-recommends libnvinfer7=7.1.3-1+cuda11.0 \
    libnvinfer-dev=7.1.3-1+cuda11.0 \
    libnvinfer-plugin7=7.1.3-1+cuda11.0

Done, no conflicts, no anything, worked better than most Medium tutorials I’ve read today.

# Reboot.

Let’s hope for the best.

UPD: no black screen, booted fine, but nvidia-smi sees no driver.

sudo apt list --installed shows all cuda stuff and nvidia driver to be installed:

nvidia-driver-465/unknown,unknown,now 465.19.01-0ubuntu1 amd64 [installed,automatic]

More worryingly, I see mentions of cuda-10-1 and cuda-11-1 together

list processes ubuntu

I should use ps axf instead of ps aux, the former gives a nice tree representation

Nvidia CUDA official installer documentation

Yet another place that makes it look easy: CUDA Toolkit 11.0 Download | NVIDIA Developer


  1. Install CUDA 10 on Ubuntu 18.04 ↩︎

  2. Install Pytorch GPU with pre-installed CUDA and cudnn - PyTorch Forums ↩︎

  3. Docker  |  TensorFlow ↩︎

  4. GPU support  |  TensorFlow ↩︎

Day 847

Docker stuff

  • Making it run as non-root: Post-installation steps for Linux | Docker Documentation
    • newgrp docker has to be run from each cli you’ll be using docker from?.. Until you restart
  • Best tutorial ever can be started with: docker run -d -p 80:80 docker/getting-started
    • It will start as docker image
    • Very readable and step-by-step
  • Docker compose
  • Random docker stop accepts the full name (distracted_perlman), but part of its container_id works!
  • Unintuitively, the COPY instruction from a Dockerfile copies the contents of the directory, but not the directory itself! 1

Clean up journalctl

Logs take space (4gb on my box!). To see how much specifically journalctl does:2

journalctl --disk-usage
sudo journalctl --vacuum-time=3d

Jupyter notebooks has terminals!

New -> Terminal. (Which you can use to access your docker running jupyter-notebook)

Docker build contexts and relative paths

$ docker build -t dt2test -f ./docker/Dockerfile . - passes the Dockerfile as explicit parameter, inside it paths are relative to the folder you run docker build in.

For docker compose:

#docker-compose.yml
version: '3.3'    
services:
      yourservice:
        build:
          context: ./
          dockerfile: ./docker/yourservice/Dockerfile

A lot of other nice options at Docker: adding a file from a parent directory - Stack Overflow


  1. Dockerfile reference | Docker Documentation↩︎

  2. 7 Simple Ways to Free Up Space on Ubuntu and Linux Mint - It’s FOSS ↩︎

Day 843

Python dataclasses

HuggingFace

“Token classification” includes but is not limited to NER: Hugging Face – The AI community building the future.. Really nice new correct phrase I’ll be using!

Installing (after tensorflow and/or pytorch):

pip install transformers

Caches by default in user folder but can be overridden:

export HF_HOME="/data/sh/experiments/bert/cache" 

The “hosted inference API” on the website is really cool! dslim/bert-base-NER · Hugging Face

Example of converting conll dataset to what BERT expects: Fine Tuning BERT for NER on CoNLL 2003 dataset with TF 2.0 | by Bhuvana Kundumani | Analytics Vidhya | Medium

The BERT model documentation shows the tokenizers etc etc etc. - BERT — transformers 4.5.0.dev0 documentation

python datasets package

Here datasets is imported: transformers/requirements.txt at master · huggingface/transformers

TODO - what is this and where can I learn more? Is this HF specific? What else is there?

HuggingFace datasets

It has a really nice interface for searching datasets! Filter by task, language, etc.

German NER datasets: Hugging Face – The AI community building the future.

Some German NER models, sometimes based on bert: Hugging Face – The AI community building the future.

Huggingface converting between tf and pytorch

Converting Tensorflow Checkpoints — transformers 4.5.0.dev0 documentation

Is this real?

export BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12

transformers-cli convert --model_type bert \
  --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
  --config $BERT_BASE_DIR/bert_config.json \
  --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin

Random / recipes / cooking

Tatar von geräuchertem Forellenfilet mit Avocado - Annemarie Wildeisens KOCHEN

Die Forellenfilets in kleine Würfelchen schneiden. Die Schalotte schälen und sehr fein hacken. Die Cherrytomaten je in 6 oder 8 Stücke schneiden. Alle diese Zutaten in eine kleine Schüssel geben und sorgfältig mit der Mayonnaise mischen.

Forelle + tomatos + mayonnaise is literally the only recipe I’ve liked with mayonnaise in it

Day 842

Jira old issue view + qutebrowser config setting

To redirect an issue to the old view, add ?oldIssueView=true.

Added this to config.py:

config.bind('<Ctrl-J>', ':open {url}?oldIssueView=true')

Ubuntu screen apt-get

(18:03:38/10185) sudo apt install screen
# ...
Suggested packages:
  byobu | screenie | iselect
The following NEW packages will be installed:

… did I just get an advert for a competitor when installing screen? :) Since when does ubuntu do this and where can I read more about it?

Day 841

Deutsch / German

“Meetingtourismus oder Papiergenerieren?” (heard at work)

Qutebrowser userscripts

It seems to run userscripts not in the virtualenv qutebrowser uses, but the standard system one? Installing packages in virtualenv didn’t work, but installing them globally did.

DVC

Moving/renaming a file/directory is easy: dvc move from to1. Automatically updates the from.dvc files. Then .gitignore and the .dvc file have to be added and committed through git as usual.

This is interesting: Data Organization — documentation

In general: Best Practices for Scientific Data Management — documentation

This guide describes Axiom Data Science’s best practices for scientific data management. The intent of these practices is to improve the accessibility and usability of your data. These practices may be followed at any time during the preparation of your dataset, but are most useful when considered at the onset of project planning and implemented during data collection.

Also related: Organising your data | Research Data Management

Tree output only directories

tree -d does it.

Git paths from root of repo

Root of repo: git rev-parse --show-prefix 2

--git-dir returns the location of the .git folder, and --show-toplevel returns the absolute location of the git root.


  1. move | Data Version Control · DVC ↩︎

  2. bash - How to get the path of the current directory relative to root of the git repository? - Stack Overflow ↩︎

Day 840

Patterns / phrases / Random

  • “It’s not a solution, but it’s an approach” - heard at work, VF

Day 839

vim delete all lines not matching pattern

I’ll memorize the g/... syntax someday.

:g!/pattern/d

I can just look for the pattern as usual with /pattern and tweak it live, then do

:g!//d

and it will atke the last used pattern.

Day 838

Pizza sauce recipes

I should try doing something more interesting with the passata di pomodoro!

Options:

In general all seem to require both tomato puree and chopped tomatoes; and olive oil + garlic + oregano/basil + (brown) sugar seems to cover 90% of cases.

Day 836

Deutsch

die Kaffeesatzleserei - reading in coffee beans (heard at work)

screen attaching screens without full name

I shouldn’t forget that screen -R screenname can be replaced by screen -R s if it’s the only screen with such a name. Not sure if better or worse than tab completion, likely worse because it’s surprising, but quite nice to use.

Logoff i3 with a CLI

i3-msg exit1 does the magic.

Blocking ips with ipset

ipset -N myset nethash  # create myset
ipset add myset 27.8.0.0/13 
iptables -I INPUT -m set --match-set myset src -j DROP # create temporary iptables thing

# making it persistent

ipset save > /etc/ipset.conf

# then enable ipset services

# Listing stuff
ipset -L

# Deleting set
ipset destroy myset

iptables basics

If you can’t destroy an ipset set because it’s being used by kernel:

iptables -L --line-numbers returns this:

Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       all  --  anywhere             anywhere             match-set myset src
...

Then to delete number 1:

iptables -D INPUT 1

Generally blocking countries

GitHub - mkorthof/ipset-country: Block countries using iptables + ipset + ipdeny.com can do both a whitelist and a blacklist.


  1. How do i suspend,lockscreen and logout? - i3 FAQ ↩︎

Day 835

Data Scientist roadmap/curriculum

Article with a very interesting graph: Becoming a Data Scientist - Curriculum via Metromap – Pragmatic Perspectives

Road to data science {:height=“500px”}

German / Deutsch

  • “Die Prioritäten sind ein bißchen volatil geworden”
  • “Sammle von XY Team ein bißchen Stimmung”

Day 832

German

der Tonus - heard at work in context of

JQ producing nice comma-separated json

Option to return objects as a list of objects (separated by a comma) · Issue #124 · stedolan/jq: TL;DR use jq "[foo]" instead of jq "foo".

Day 831

Yunohost full app information / data / install paths

yunohost app info -f appname returns the A LOT of info about the appname, including installation paths.

Qutebrowser userscripts folder location / Writing informative error messages

… can be located in ~/.config/qutebrowser/userscripts, not just in ~/.local ..! When tried to run one it didn’t find it helpfully outputted all the paths it looks for them - which is great and I’ll steal this. If a file is not found you know the person will probably need this, especially if they are many.

GNU Stow for dotfiles management

One of the cooler solutions I’ve seen: Managing dotfiles with GNU stow - Alex Pearce (There seems to be a canonical page1 I found first, but I like the other one more)

TL;DR create a directory for the dotfiles, with each folder containing dotfiles mirroring the usual dotfiles' locations in the system; Then from inside the main dotfiles directory do stow vim bash whatever and it’ll magically put it in the right place in the home directory.

This works because

Stow assumes that the contents of the

you specify should live one directory above where the stow command is run, so having our .dotfiles directory at ~/.dotfiles means using stow to manage our dotfiles just works. 2

This is awesome because:

  • No manual symlinking
  • Dotfiles directory can be easily backed up with git or whatever

The same article2’s sample github repo: dotfiles/neovim at master · alexpearce/dotfiles

Cool dotfile ideas

The stow linked github repo’s dotfiles are actually fascinating: alexpearce/dotfiles: My dotfiles.

dotfiles/.gitconfig at master · alexpearce/dotfiles:

# Clone git repos with URLs like "gh:alexpearce/dotfiles"
[url "https://github.com/"]
  insteadOf = "gh:"
[url "git@github.com:"]
  pushInsteadOf = "gh:"
# Clone CERN GitLab repos with URLs like "gl:lhcb/Hlt"
[url "ssh://git@gitlab.cern.ch:7999/"]
  insteadOf = "gl:"

Git config aliases

Applying the above to my own configs in ~/.gitconfig.

Assuming the ssh port is 1234 ~/.gitconfig is like

[url "ssh://git@myserver:1234/"]
  insteadOf = "gh:"

and then in the per-repo settings something similar to

[remote "bitbucket"]
	url = gh:myusername/myproject.git

Cloning it is now easy:

git clone gh:myusername/myproject

Neat!

Jekyll syntax highlighting supported languages

List of supported languages and lexers · rouge-ruby/rouge Wiki Quite a lot! Will try the generic conf for the .gitconfig above.


  1. Brandon Invergo - Using GNU Stow to manage your dotfiles↩︎

  2. Even better description than the canonical page: Managing dotfiles with GNU stow - Alex Pearce ↩︎

Day 830

Yunohost

I’m very impressed by it! Makes everything really easy, I remember the last time I had to install stuff manually. After 48h 9/10, some things surprised me (removing root ssh access…) but they were always mentioned in the relevant docu I hadn’t read.

Official docu is quite okay, but rarely appeared when I was googling my problems. My instinct is to Google the problem instantly - sometimes they should actually be to find and check any existing official documentation/README first, then google. (An even better instinct would be to skim any official documentation before starting, as religiously as I do it for unknown real-life 3D things.)

Adding subdomains for Yunohost

This took me too long to find, has info about correct DNS records: DNS and subdomains for the applications | Yunohost Documentation

By trial and error the complete process is:

  1. Add DNS record for subdomain like last examples here:
    @         A            XYZ.XYZ.XYZ.XYZ
    @         AAAA         1234:1234:1234:FFAA:FFAA:FFAA:FFAA:AAFF
    *         CNAME        mydomain.com.
    agenda    CNAME        mydomain.com.
    blog      CNAME        mydomain.com.
    rss       CNAME        mydomain.com.
    
    ``
  2. Add new domain to yunohost, input the domain with subdomain (subdomain.my.domain) as it if were new
  3. Do a diagnostic, which does DNS checks too, which are needed for Letsencrypt
  4. Install letsencrypt certificate from the usual Yunohost panel

I kept messing up NAME and DATA of the CNAME records because I was following more the other ones Yunohost created, a row of

Name: xmpp-upload.my.domain
Data: @

For subdomainname.my.domain I needed this (kinda-sorta-reversed from the above; as usual, dots are significant):

Name: my.domain.
Data: subdomainname

Random / colored fonts generator / CLI

cfonts is like figlet, but with many more settings (colors and alignment blew my mind!)! Link has a lot of colorful examples. I might get a nice colorful motd and/or banner soon. :)

Setting a new hostname linux

There’s a command for that: hostnamectl set-hostname new-hostname

I like the idea of having ~/.local/bin in my $PATH, and putting there symbolic links (ln -s TARGET LINK) to my usual folder where I have programs/executables. I’d even have a separate thing in $PATH for shell scripts and binaries, which will get rid of so many stupid CLI aliases I have whose function is to point to a single executable with a long path. TODO - look at my aliases and commands I run often and see how many of them can I symlink

Day 829

VPS plans

  • Taskwarrior sync
  • git for ~/.timewarrior/ and similar folders
  • git for dotfiles
  • Some basic automated backups of small important things
  • Possibly some Telegram bots will live there
  • CalDAV & Contacts sync - both for sync and for backups
  • Possibly self-hosted password management?

Timewarrior on-modify hook for taskwarrior

Had always problems with umlauts etc, looked at the source, changed #!/usr/bin/env python to #!/usr/bin/env python3 - now it works! Wanted to do a pull request, but it’s fixed on github master1, the apt repo has an older version as it often does.

git clone to different directory

.. As expected. git clone git@what:ever outputdirectory. git clone git@what:ever . works.

Setting up serhii.net

New domain, yay! I’ll slowly move stuff there, starting with this diensttagebuch.

Setting up multiple remotes in github + .git/config

I wanted to set up two remotes, so that the dtb deploy.sh script after building the html & rsync-ing would push it to both the github dtb repo and my own. Followed this basically (except that I had deleted origin by error in the process, so recreated it back again and added both remotes to it so I’ll still be able to do git push origin master): How to push to multiple git remotes at once. Useful if you keep mirrors of your repo..

Mostly copying from there, changing/sanitizing some of my configs:

# Assume the git repost are set up like this
git remote add github git@github.com:muccg/my-project.git #this is the one "origin" pointed to to
git remote add bb git@bitbucket.org:ccgmurdoch/my-project.git

# Add to origin two remote urls for push
git remote set-url --add --push origin git@github.com:muccg/my-project.git
git remote set-url --add --push origin git@bitbucket.org:ccgmurdoch/my-project.git

# Look at the result
git remote show origin

which outputs this:

> git remote show origin
* remote origin
  Fetch URL: git@github.com:pchr8/my-project.git
  Push  URL: git@bitbucket.org:pchr8/my-project.git
  Push  URL: git@github.com:pchr8/my-project.git
  HEAD branch: master

Mentioned in the comments, it works, but has to be done twice of as it seems to rewrite the original remote: git remote set-url --add --push origin <...>

But maybe the most interesting thing there is .git/config! I didn’t know it existed, it shows most of the same things but much easier to read/edit! It currently shows something like this:

> cat  .git/config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[branch "master"]
[user]
	email = me@me.me
	name = SH
[remote "bb"]
	url = git@bitbucket.org:pchr8/my-project.git
	fetch = +refs/heads/*:refs/remotes/bb/*
	pushurl = git@bitbucket.org:pchr8/my-project.git
[remote "github"]
	url = git@github.com:pchr8/my-project.git
	fetch = +refs/heads/*:refs/remotes/github/*
	pushurl = git@github.com:pchr8/my-project.git
[remote "origin"]
	url = git@github.com:pchr8/my-project.git
	fetch = +refs/heads/*:refs/remotes/origin/*
	pushurl = git@bitbucket.org:pchr8/my-project.git
	pushurl = git@github.com:pchr8/my-project.git

Creating redirects to new website

Adding the RedirectPermanent lines to .htaccess in the root of pchr8.net, that now contains the following:

ErrorDocument 404 /404.html
ErrorDocument 403 /404.html
ErrorDocument 500 /500.html

RewriteRule ^wiki/(.*)$ /f/$1 [R=301,NC,L]
RewriteRule ^fiamma/(.*)$ /f/$1 [R=301,NC,L]

RedirectPermanent /d/dtb https://serhii.net/dtb
RedirectPermanent /blog https://serhii.net/blog

Experimenting with rewriting everything except /f/, seems to work except for the main page https://www.pchr8.net/f/index.php/Pchr8.net_wiki_thing

RewriteEngine on

#RewriteRule (f) - [L]
RewriteCond %{REQUEST_URI} !^/f
RewriteRule (.*) https://serhii.net/$1 [R=301,L]

It gets redirected to serhii.net - maybe it chokes on the many weird characters or the repeat of pchr8.net?..

Setting up HTTPS/TLS for serhii.net

As per nfs docs 2, it’s very easily done just by running YourPrompt> tls-setup.sh, and nfs takes care of all autorenewals, automatically sets up redirects etc. Awesome!

utimer

utimer can do a countdown, count-..up?, and can work as a stopwatch. It outputs time remaining too.

English

A pizza dough recipe3 reminded me that

DTB/markdown/footnotes/macro improvement idea

I have my vim macro for footnotes where it creates the [^..] things and then I paste the URI manually, but what I’d actually like is something that automatically creates a footnote at current cursor position, and as content uses the URI currently in the clipboard register! TODO (And also try to make it readable/interpretable this time)

Yunohost

To create a subdomain, you have to add it as “new” new domain and it takes care of everything, no magic with DNS records needed


  1. timewarrior/on-modify.timewarrior at dev · GothenburgBitFactory/timewarrior ↩︎

  2. FAQ - NearlyFreeSpeech.NET ↩︎

  3. Easy Homemade Pizza Dough - JoyFoodSunshine ↩︎

Day 825

taskwarrior non-work user account

Changed the zsh alias for it:

s () {task s project.not:w sprint.not:s "$*"}

Now on my non-work account, it shows non-work tasks from any sprint except “s” (which is a proxy of due:someday).

German foreign words

Foreign Words (Fremdwörter) - really nice! Has specific suffixes and what genders they create in German. In general - I remember that excellent website.

Also: “das Thema, die Themen”) - which plural rule is that? TODO

DTB - TODO

Given that I need to push/pull it a lot now, I should exclude the generated .html files in .gitignore

qutebrowser

W opens the last closed window! … on the topic of ‘learn well the tools you use daily’

ding

Installed ding! Still remains the best dictionary program ever. ding buch works!

TODO - add keybinding to search for currently selected word. Or a basic prompt to quickly look for words, a la dtb - and that ideally adds the needed words to a list, and maybe even generates anki flashcards from them!

ding -m to start it minimally, likely make it floating for i3 by class, is a really nice start. Added this to config:

## Ding float
bindsym $ms+Shift+d exec ding -m
for_window [class="Ding"] floating enable

(got class from xprop)

Redshift settings for late-night work

If default automatic settings are too strong, these work well: redshift -xO 2500 -b 0.7

Day 823

Noisetorch / polkit / policykit / pkexec saga

Couldn’t load noisetorch, error 127 when attempting to get the needed privileges. The help of Noisetorch said this means pksudo doesn’t work, and to fix this. After some googling, found a solution:

apt install policykit-1-gnome

Then add /usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1 & to your autostart configuration. 1


  1. Debian User Forums • View topic - [SOLVED] Problem with polkit in testing MATE ↩︎

Day 821

Interactive mode matplotlib

According to the docu it should be this, not working for me:

plt.ion()

Somehow it magically worked before without any changes from my side actually. Anyway, this1 worked:

import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt

  1. python - Interactive plotting in Pycharm debug console through matplotlib - Stack Overflow ↩︎

Day 818

i3 startup - final

I can’t start everything from within i3 config. keynav doesn’t work (though it’s running), and compton creates a black strip in the bottom monitor when started as exec compton via i3. Though executing a startup script from within i3, a script starting everything else I need, somehow works. I remember dealing with this in the past, and this created the current chaotic setup.

Startup script (./s/s.sh) :

setxkbmap -option -option 'grp:rctrl_toggle, compose:rwin, compose:paus' v5,ruua
xmodmap ~/s/mod4
xcape -e 'Control_L=Escape' -t 100 

autorandr -l home

feh --bg-center ~/s/bg.jpg ~/s/bg.jpg

compton

keynav

i3 config startup script:

exec ~/s/s.sh
exec --no-startup-id redshift
exec --no-startup-id nm-applet

Removing dysfunctional setups

vim - remove save as root

I had this, but it started too often by error.

:W sudo saves the file 
" command W w !sudo tee % > /dev/null

New zsh prompt

Added this in a modified sh-trapd00r theme:

dir_status="%{$c1%}%* %B%7c/ %?"
PROMPT='%{$fg_bold[green]%}%p%{$reset_color%}${dir_status} ${ret_status}%{$reset_color%}
%{$fg_bold[green]%}> %{$reset_color%}'

Day 817

loginctl as a way to manage sessions of logged in users

Instead of killing all processes belonging to someone, loginctl will return all sessions, and loginctl kill-session $number will log the user off!

New non-work user account!

Set my old Lain background with feh. I should look at some of my old i3 settings etc, to make it look different from the work one.

  • zsh theme: trapd00r
  • vim theme: pablo

General plans for vacation

  • Don’t touch any ‘should’s - java, python, …; mostly focus on ‘housekeeping’ things if I want to do stuff with the computer
  • Learn to use kitty well
    • Highlighting/copying URIs etc especially
    • See if I can use it to replace some of my hacks
  • Learn to use tmux or screen well
    • Screen is available almost everywhere, but tmux is ‘better’
  • Learn to use vim much better
    • Make an effort to learn it systematically and well from the beginning, I have a lot of antipatterns
    • w/E etc
    • vim recovery of swap files
  • Sort out my i3/user config and all configs in general
    • Something easy to enable/disable keyboards (xinput float ..)
    • Something to turn on/off audio/webcamera
    • Move container splitting keybinding further away from window closing keybinding
  • Sort out all dotfiles
    • A place where they are by default and can be imported/overwritten
    • In general any kind of dotfiles management/backup
  • Sort out my startup scripts
    • In general something that doesn’t make me afraid to disconnect the laptop from the screen
      • Automatically use connected screens, without arandr-ing every time
      • Same for keyboards
      • Same for keyboard layouts
      • Run redshift and stuff only once
      • Even just an i3 keybinding that sets up what’s needed

“Ricing” - English / Unix / …

  • Ricing is “making improvements to a system that don’t actually do anyone any good, and can sometimes have negative ramifications” 1
  • “Rice” is “a word that is commonly used to refer to making visual improvements and customizations on one’s desktop. It was inherited from the practice of customizing cheap Asian import cars to make them appear to be faster than they actually were” 2

(was curious about the name of a PPA)

i3 stuff

Test config file:

displays:
  - name: eDP-1
    workspaces: [1, 0]
    randr_extra_options: "--primary --mode 2560x1440"
  - name: HDMI-2
    workspaces: [2, 3, 4]
    randr_extra_options: "--above eDP-1"

autorandr for flexible multimonitor setup

This is even better than the above: phillipberndt/autorandr: Auto-detect the connected display hardware and load the appropriate X11 setup using xrandr It saves configs readably and automatically to ~/.config/autorandr/config

General small things

  • autorandr set up
  • i3lock
  • better autostart
    • start the diensttagebuch, work notes in the correct workspaces
    • start slack, Telegram and co in the correct workspaces
    • Put workspaces on the correct screens

i3-gaps fun

Very simple config:

gaps inner 10
gaps outer 10

Installed compton to get transparent terminals. Added this to kitty config:

background_opacity 0.8

Git use use specific public key file

When using public key and ssh for git, when you can’t use ssh-add ..., this works: GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa_example" git clone example 3


  1. Urban Dictionary: ricing ↩︎

  2. themeing/dictionary - unixporn ↩︎

  3. ssh - How to tell git which private key to use? - Super User ↩︎

Day 813

Pycharm / matplotlib / pyplot debugging

I can happily use plt.plot()/plt.imshow() inside the <Alt-F8> and debugger console windows, it’ll be shown!

Recursively change owner in files owned by other user in current directory

Replace -user root with source user, $USER expands to user currently running command:

sudo find ~ -type d -user root -exec sudo chown -R $USER: {} +

Day 812

sshfs / ‘Transport endpoint not connected’

In line with Day 784 about unmounting broken endpoints, yesterday I got a lot of errors (thunar didn’t start, I blamed memory, but df -h also didn’t start…), at the end the issue was with a sshfs directory:

fuse: bad mount point ./mountpoint': Transport endpoint is not connected`

Using day 784 didn’t help, still got the above error. This helped: fusermount -uz myserver

Also, TODO: Why doesn’t linking stuff like this work?

{%raw%}
[Day 784]({% post_url 2021-02-23-day784.markdown %})
{%endraw%}

numpy true booleans

a is True is false for a numpy array of one element a, even if it’s value is True. a == True works correctly. Why does this happen?

pycharm debugging in console

You can use the console not just to look for output, but to interact with the variables etc! Why didn’t I think of this before: Using Debug Console | PyCharm

Day 811

OpenCV documentation

I like giving code examples in C++, Java and Python for the same help topic! OpenCV: Creating Bounding boxes and circles for contours

Disabling touchpad while typing (xinput)

(22:31:53/11773)~/$ xinput list-props 15
Device 'SynPS/2 Synaptics TouchPad':
	Device Enabled (170):	1
	Coordinate Transformation Matrix (172):	1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
	Device Accel Profile (304):	1
	Device Accel Constant Deceleration (305):	2.500000
	Device Accel Adaptive Deceleration (306):	1.000000
	Device Accel Velocity Scaling (307):	12.500000
	Synaptics Edges (327):	1574, 5368, 1408, 4444
	Synaptics Finger (328):	25, 30, 0
	Synaptics Tap Time (329):	180
	Synaptics Tap Move (330):	248
	Synaptics Tap Durations (331):	180, 180, 100
	Synaptics ClickPad (332):	1
	Synaptics Middle Button Timeout (333):	0
	Synaptics Two-Finger Pressure (334):	282
	Synaptics Two-Finger Width (335):	7
	Synaptics Scrolling Distance (336):	112, 112
	Synaptics Edge Scrolling (337):	1, 0, 0
	Synaptics Two-Finger Scrolling (338):	1, 0
	Synaptics Move Speed (339):	1.000000, 1.750000, 0.035417, 0.000000
	Synaptics Off (340):	0
	Synaptics Locked Drags (341):	0
	Synaptics Locked Drags Timeout (342):	5000
	Synaptics Tap Action (343):	2, 3, 0, 0, 1, 3, 0
	Synaptics Click Action (344):	1, 3, 0
	Synaptics Circular Scrolling (345):	0
	Synaptics Circular Scrolling Distance (346):	0.100000
	Synaptics Circular Scrolling Trigger (347):	0
	Synaptics Circular Pad (348):	0
	Synaptics Palm Detection (349):	0
	Synaptics Palm Dimensions (350):	10, 200
	Synaptics Coasting Speed (351):	20.000000, 50.000000
	Synaptics Pressure Motion (352):	30, 160
	Synaptics Pressure Motion Factor (353):	1.000000, 1.000000
	Synaptics Resolution Detect (354):	1
	Synaptics Grab Event Device (355):	0
	Synaptics Gestures (356):	1
	Synaptics Capabilities (357):	1, 0, 0, 1, 1, 1, 1
	Synaptics Pad Resolution (358):	54, 45
	Synaptics Area (359):	0, 0, 0, 0
	Synaptics Soft Button Areas (360):	3471, 0, 4054, 0, 0, 0, 0, 0
	Synaptics Noise Cancellation (361):	28, 28
	Device Product ID (297):	2, 7
	Device Node (296):	"/dev/input/event5"
(22:31:59/11774)~/$ xinput set-prop 15 349 1

Day 807

Google Hangouts highlighting people

If there are too many people with video on, Google Hangouts moves the ones who talk closer to the beginning, making them visible?

Day 805

pycharm/intellij running config environment variables spaces

Got bitten yet again when copypasting them - the name of one of them had four leading tabs. THAT DIDN"T GET SHOWN UNTIL I TRIED TO EDIT THE ENVIRONMENT VARIABLE IN THE PYCHARM WINDOW - it removes them when visualizing. Why? (The parameter of the last one had a trailing space too)

Python negative 0

-0.0 exists as float, and gets stored like this. Though it’s not less than 0 or +0.0. Can’t easily google a way to detect if it’s a negative 0 or not.

Day 803

Signature detection

Random / CLI / CLI task manager / replacement for screen/tmux

GitHub - Nukesor/pueue: Manage your shell commands. (thank you AA)

Day 801

Naming cheatsheet

GitHub - kettanaito/naming-cheatsheet: Comprehensive language-agnostic guidelines on variables naming. Home of the A/HC/LC pattern. (thank you AA)

From it:

Name Prefix Action (A) High context (HC) Low context (LC)
getUser get User
getUserMessages get User Messages
handleClickOutside handle Click Outside
shouldDisplayMessage should Display Message

Day 800

Detectron2 dataloader training in parallel num_workers (“process exited unexpectedly”)

When training on different GPUs on the same server, I get errors like RuntimeError: DataLoader worker (pid 30141) exited unexpectedly with exit code 1.

The fix was to set the number of workers to 0: 1

cfg.DATALOADER.NUM_WORKERS = 2

  1. Runtime Error with DataLoader: exited unexpectedly · Issue #5301 · pytorch/pytorch · GitHub ↩︎

Day 797

Object detection / segmentation metrics & evaluation

From SO: 1

[..]the only difference between mAP for object detection and instance segmentation is that when calculating overlaps between predictions and ground truths, one uses the pixel-wise IOU rather than bounding box IOU.

ROC curve / cutoff point

Finding an optimal cutoff point in a ROC curve is largely arbitrary (or ‘depending on what you need’ based on the actual thing). A lot of ways to find this. (Nice list here, but I’d see if I can find a paper with a good overview: data visualization - How to determine best cutoff point and its confidence interval using ROC curve in R? - Cross Validated)

Detectron2 internals

Nice series of posts on how Detectron2 works inside: Digging into Detectron 2 — part 1 | by Hiroto Honda | Medium

Paper with object detection metrics comparison with the focus on COCO & open source

Electronics | Free Full-Text | A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit

Untderstanding model performance by looking at examples it got wrong but was confident about

From How to work with object detection datasets in COCO format | by Eric Hofesmann | Feb, 2021 | Towards Data Science:

The best way to build intuition about how your model performs is by looking at predictions that it was confident about but got wrong. With FiftyOne, this is easy. For example, let’s create a view into our dataset looking at the samples with the most false positives

More examples of the same: IoU a better detection evaluation metric | by Eric Hofesmann | Towards Data Science


  1. neural networks - Why is mAP (mean Average Precision) used for instance segmentation tasks? - Cross Validated ↩︎

Day 796

Notes bullet points

In my text notes, I use indentation heavily, but use bullet-point-dashes (-) and just indentation almost interchangeably:

One two
	Three
	Four 
	Five
		- six
		- seven
		- eight
			Nine
			Ten
	- 12
	- Thirteen

Next part

From now on:

  • Indentation to signal thematic shifts / logical blocks / things following each other chronologically
  • Bullet points for lists and list-like things, where order doesn’t matter

Day 794

Pytorch access GPU tensors from memory

tensor.cpu().numpy() needs to be done when using GPU.

Random / cooking

Паста с морепродуктами в сливочном соусе рецепт – итальянская кухня: паста и пицца. «Еда»

Nvidia tool for GPU/CPU optimization

NVIDIA Nsight Systems | NVIDIA Developer

Found here (a nice article too): Object Detection from 9 FPS to 650 FPS in 6 Steps | paulbridger.com

Pytorch multiprocessing

Multiprocessing best practices — PyTorch 1.8.0 documentation

TL;DR:

torch.multiprocessing is a drop in replacement for Python’s multiprocessing module

Day 792

Detectron2 run without GPU

If Detectron2 complains about wanting a GPU and finding no CUDA (because there’s none), the script can be set to CPU-only through the settings:

cfg.MODEL.DEVICE = 'cpu'

Detectron2 instances

I should read documentation more often: detectron2.structures — detectron2 0.3 documentation

  • They can be indexed as a mask:
	category_3_detections = instances[instances.pred_classes == 3]
	confident_detections = instances[instances.scores > 0.9]

In general about model outputs: Use Models — detectron2 0.3 documentation

Pytorch converting Tensor to floats

mytensor.numpy() is unsurprisingly easy.

Shapely prepared geometry operations

Shapely geometries can be processed into a state that supports more efficient batches of operations.

(The Shapely User Manual — Shapely 1.7.1 documentation)

Shapely find out if something is a multipolygon:

if joined_boxes.geom_type == 'MultiPolygon': is much cleaner than the isinstance(joined_boxes, MultiPolygon) I’ve been using!

Also - TODO - why is a Polygon that created a MultiPolygon within() it, if `within()..

Returns True if the object’s boundary and interior intersect only with the interior of the other (not its boundary or exterior).

Their boundary should touch, so shouldn’t be valid?

R-tree spatial indexing

Nice (and one of the only..) graphic explanation: R-tree Spatial Indexing with Python – Geoff Boeing

Shapely has a partial implementation: 1

Pass a list of geometry objects to the STRtree constructor to create a spatial index that you can query with another geometric object. Query-only means that once created, the STRtree is immutable.

TL;DR:

tree = STRtree(all_geoms)
results = tree.query(query_geom)

In general if I’ll be working more with shapes I should hang out in GIS places to to absorb approaches and terminology. One of R-Tree’s use-cases is say “find restaurants inside this block” which can also be solved by blind iteration (but shouldn’t).

qutebrowser yank selection

Finally got the more familiar keybinding to work, as usual config.py:

config.bind('<Ctrl-Shift-C>', 'yank selection')`
config.bind(',y', 'yank selection')

Python dependencies list

johnnydep2 is really cool and visualizes the dependencies of something without installing them (but still downloads them!)

Trash and disk space

Found .local/share/Trash with 33Gb of ..trash in it.

Python dependencies wheel

A .whl file is just an archive, can be unzipped. The entire list of dependencies is in yourpackage.dist-info/METADATA, looks like this:

Requires-Python: >=3.6
Provides-Extra: all
Provides-Extra: dev
Requires-Dist: termcolor (>=1.1)
Requires-Dist: Pillow (>=7.1)

  1. The Shapely User Manual — Shapely 1.7.1 documentation ↩︎

  2. wimglenn/johnnydep: Display dependency tree of Python distribution ↩︎

Day 790

python3.7

..exists, and in general I should pay more attention to the new python versions and their changes.

tiffsplit

Ubuntu Manpage: tiffsplit - split a multi-image TIFF into single-image TIFF files

Installs as libtiff-tools, basename can be used as prefix.

Day 789

Inkscape joining (union) of paths

When joining/adding two paths (as in discrete math union) located in different layers, the resulting path will be located in the layer selected when doing the joining.

Inkscape groups

.. are recursive! Grouping two groups works; ungrouping them leads the original two groups!

Day 787

Python multiprocessing/threading basics

From Multiprocessing vs. Threading in Python: What Every Data Scientist Needs to Know

Terminology:

  • Processes: instances of a program being executed; don’t share memory space

    • Slower to create, take a bit more memory and stuff
  • Threads: components of a process that run in parallel; share memory, variables, code etc.

    • Faster to create, less overhead
    • Much easier to share objects between them
  • Race Condition: “A race condition occurs when multiple threads try to change the same variable simultaneously.” (Basically - when order of execution matters)

  • Starvation: “Starvation occurs when a thread is denied access to a particular resource for longer periods of time, and as a result, the overall program slows down.”

  • Deadlock: A deadlock is a state when a thread is waiting for another thread to release a lock, but that other thread needs a resource to finish that the first thread is holding onto.

  • Livelock : Livelock is when threads keep running in a loop but don’t make any progress.

Python / GIL

In CPython, the Global Interpreter Lock (GIL) is a (mutex) mechanism to make sure that two threads don’t write in the same memory space.

Basically “for any thread to perform any function, it must acquire a global lock. Only a single thread can acquire that lock at a time, which means the interpreter ultimately runs the instructions serially.” Therefore, python multithreading cannot make use of multiple CPUs; multithreading doesn’t help for CPU-intensive tasks, but does for places where the bottleneck is elsewhere - user interaction, networking, etc. Multithreading works for places w/o user interaction and other bottlenecks where the tasks are CPU-bound, like doing stuff with numbers.

Tensorflow uses threading for parallel data transformation; pytorch uses multiprocessing to do that in the CPU.

TODO - why does Tensorflow do that?

Python libraries

Python has two libraries, multithreading and multiprocessing, with very similar syntax.

Comparing execution time

Both pictures from the same article above1:

  • One process is slower than one thread always; for more than one, processes win for CPU-only tasks, threads for bottlenecked tasks.
  • More processes than cores doesn’t improve life by much in any case (still better than the same amount of threads though); in the picture, there are four cores.

Python-specific points

  • Easier to make errors in multithreading programs (easier to share data, but you have to keep in mind object synchronisation and race conditions).
  • Threads can’t do true parallelism in Python due to GIL
  • The OS schedules processes, Python schedules threads
  • “Child processes are interruptible and killable, whereas child threads are not. You have to wait for the threads to terminate or join.”

For data science

  • Reading data from disk is I/O bound => multithreading
  • Calculating stuff on CPU/GPU is CPU bound => multiprocessing
  • Storing results => multithreading

Concurrency / parallelism / Python

From Python Multi-Threading vs Multi-Processing | by Furqan Butt | Towards Data Science:

Concurrency is essentially defined as handling a lot of work or different units of work of the same program at the same time.

Doing a lot of work of the same program at the same time to speed up the execution time.

Parallelism has a narrower meaning.

Python - concurrent.futures for multithreading and multiprocessing

Multithreading:

import concurrent.futures
with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(function_name, iterable)

This would create a thread for each element in iterable.

Multiprocessing works in an extremely similar way:

import concurrent.futures
with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(function_name, iterable)

More about it, as usual, in the docs:

The asynchronous execution can be performed with threads, using ThreadPoolExecutor, or separate processes, using ProcessPoolExecutor. Both implement the same interface, which is defined by the abstract Executor class. 2

Questions

Does concurrent.futures have any tradeoffs compared to doing multiprocessing.Pool() like the following?

pool = multiprocessing.Pool()
pool.map(multiprocessing_func, range(1,10))
pool.close()

Measuring and reading time

Python parallelism example

Parallelising Python with Threading and Multiprocessing | QuantStart has a nice point:

time python thread_test.py

real    0m2.003s
user    0m1.838s
sys     0m0.161s

Both user and sys approximately sum to the real time. => No parallelization (in the general case). After they use multiprocessing, two processes, real time drops by two, while user/sys time stays the same. So time on CPU per second is the same, but we have two CPUs that we use, and we get real time benefits.

Reading and interpreting time output:

Excellent article, copying directly: Where’s your bottleneck? CPU time vs wallclock time

real: the wall clock time. user: the process CPU time. sys: the operating system CPU time due to system calls from the process.

In this case the wall clock time was higher than the CPU time, so that suggests the process spent a bunch of time waiting (58ms or so), rather than doing computation the whole time. What was it waiting for? Probably it was waiting for a network response from the DNS server at my ISP.

Important: If you have lots of processes running on the machine, those other processes will use some CPU.

Reading CPU time ratios

Directly copypasting from the article above, “CPU” here is “CPU Time” (so user in the output of the command), second is “real” (=wall; real-world) time.

If this is a single-threaded process:

  • CPU/second ≈ 1: The process spent all of its time using the CPU. A faster CPU will likely make the program run faster.
  • CPU/second < 1: The lower the number, the more of its time the process spent waiting (for the network, or the harddrive, or locks, or other processes to release the CPU, or just sleeping). E.g. if CPU/second is 0.75, 25% of the time was spent waiting.

If this is a multi-threaded process and your computer has N CPUs and at least N threads, CPU/second can be as high as N.

  • CPU/second < 1: The process spent much of its time waiting.
  • CPU/second ≈ N: The process saturated all of the CPUs.
  • Other values: The process used some combination of waiting and CPU, and which is the bottleneck can be harder to tell with just this measurement.

A bit more about cpu time

  • The user-cpu time and system-cpu time [..] are the amount of time spent in user code and the amount of time spent in kernel code. 3
  • multi-core machines and multi-threaded programs can use more than 1 CPU second per elapsed second 3

Python-specific thread programming:

def thread_task(lock): 
    """ 
    task for thread 
    calls increment function 100000 times. 
    """
    for _ in range(100000): 
        lock.acquire() 
        increment() 
        lock.release() 

  1. Multiprocessing vs. Threading in Python: What Every Data Scientist Needs to Know ↩︎

  2. concurrent.futures — Launching parallel tasks — Python 3.9.2 documentation↩︎

  3. What specifically are wall-clock-time, user-cpu-time, and system-cpu-time in UNIX? - Stack Overflow ↩︎

Day 786

ELIZA chatbot source

This is the script of the DOCTOR program for ELIZA: eliza/doctor.txt at master · wadetb/eliza

SSH port forwarding - you can forward multiple ports!

The -L option can be specified multiple times within the same command. Every time with different ports. 1

Here’s an example:

ssh me@remote_server -L 8822:REMOTE_IP_1:22 -L 9922:REMOTE_IP_2:22

And an even better solution from there, adding this to ~/.ssh/config

Host port-forwarding
  Hostname remote_server
  User me
  LocalForward 6007 localhost:6007
  LocalForward 6006 localhost:6006
  Port 10000

and then just do ssh pf!

Latex color list

A list of all colors in latex supported via the various packages: color - Does anyone have a newrgbcolor{colourname}{x.x.x} list? - TeX - LaTeX Stack Exchange


  1. ssh -L forward multiple ports - Stack Overflow ↩︎

Day 785

Jupyter notebook - show token

Pressing <Ctrl-C> in a Terminal where jupyter-notebook is running will show a list of running kernels/notebooks, which will include the token:

1 active kernel
Jupyter Notebook 6.2.0 is running at:
http://localhost:6007/?token=3563b961b19ac50677d86a0952c821c2396c0255e97229bc
 or http://127.0.0.1:6007/?token=3563b961b19ac50677d86a0952c821c2396c0255e97229bc

mAP (mean average precision) metric

Nice description: Measuring Object Detection models - mAP - What is Mean Average Precision?

TL;DR a way to uniformly calculate results of object detection over an entire dataset, accounding for different thresholds (“my 50% confidence is your 80%). We get such thresholds that recall is 0.1, 0.2, …, 1.0 and then measure precision at these points; take the mean.

A bit more details: rafaelpadilla/Object-Detection-Metrics: Most popular metrics used to evaluate object detection algorithms.

Day 784

Force unmount / umount

One can use mount without arguments to get the list of mounted filesystems! o

Killing anything that uses a directory:1

fuser -kim /address  # kill any processes accessing file
unmount /address

(-k is kill, -i is “ask nicely before killing”)

Reproducibility / configs / experiments / yacs

rbgirshick/yacs: YACS – Yet Another Configuration System is a “lightweight library to define and manage system configurations, such as those commonly found in software designed for scientific experimentation”. It’s used by detectron2, serializes configs in yaml files. Nicely supports standard settings and experiment overrides and CLI overrides. Basically what I’ve been trying ot hack together in some of my scripts.

Detectron2 error with test set when none set.

Got: FileNotFoundError: [Errno 2] No such file or directory: 'datasets/coco/annotations/instances_val2017.json at the end of trainings.

Solution was to have cfg.DATASETS.TEST = () explicitly set, not commented out like I had. 2

so it’s a mystery why cfg.DATASETS.TEST is looking for datasets/coco/annotations/instances_val2017.json

Indeed.

Detectron2 evaluation

Example of how to use EvalHook to run functions: detectron2/train_net.py at master · facebookresearch/detectron2 (but I’d like to implement the eval as a subclass)


  1. linux - How to unmount a busy device - Stack Overflow ↩︎

  2. Training on custom datasets, but still looking for ‘datasets/coco/annotations/instances_val2017.json’ · Issue #2012 · facebookresearch/detectron2 ↩︎

Day 783

Python to read

Python path / pathlib

The python3 way to work with paths seems to be pathlib — Object-oriented filesystem paths — Python 3.9.2 documentation, not the old os.path.*

Split is Path (close to really-existing things), and PurePath - abstract paths, without connection to any real filesystem.

Day 779

Python working with shapes

Shapely is awesome! And easy to play with in jupyter notebook

SSH port forwarding for tensorboard/jupyter

To access a Tensorboard (..or anything) running on a remote server servername on port 6006: ssh -L 6006:127.0.0.1:6006 me@servername

After this, tensorboard is bound to the local port 6006, so 127.0.0.1:6006.

Tensorboard has to be run with --host=127.0.0.1 to make it accessible from outside.

Jupyter - the link with the token can simply be followed (or copypasted), if the port is the same in both localhost and server.

Day 777

matplotlib/pyplot invert/reverse axis

Unsurprisingly intuitive:

ax.set_ylim(1, 0)

(of course, problematic if you don’t know your actual limit)

EDIT Mi 10 Mär 2021 19:23:20 CET: There’s an even better solution! 1

ax.invert_yaxis()

Install pytorch on CUDA 10.0 + verify torch/cuda installation

Pytorch officially doesn’t do CUDA 10.0.x, but I found this, worked perfectly: How to Install PyTorch with CUDA 10.0 - VarHowto

Installing: pip install torch==1.4.0 torchvision==0.5.0 -f https://download.pytorch.org/whl/cu100/torch_stable.html

Testing installation and GPU:

import torch
x = torch.rand(5, 3)
print(x)

torch.cuda.is_available()

  1. matplotlib.axes.Axes.invert_yaxis — Matplotlib 3.3.4 documentation ↩︎

Day 776

Dotfiles over multiple servers

Nice discussion: How do you manage your dotfiles across multiple and/or new developer machines? - DEV Community

This article also provides a really nice explanation of the general practice that many people seem to be taking: store dotfiles in GitHub, and then install them via a simple script that symlinks files and runs any additional init logic.

Day 773

NewPipe youtube music

… not that I’ve ever used it or plan to (google, don’t ban me before I finished switching to FastMail!), but - NewPipe supports searching and playing videos from Youtube Music!

Serial-position effect (memory)

Serial-position effect “is the tendency of a person to recall the first and last items in a series best, and the middle items worst”. Related is the Von Restorff effect about the most different stimuli being easier to remember.

Day 772

Setting up the touchpad

.. never used it because didn’t find it pleasant to use because no scrolling and clicking as I’m used to, but I can fix this! Google told me I should install synaptics stuff and use synclient to config it, but..

(21:30:13/11094)~/$ synclient
Couldn't find synaptics properties. No synaptics driver loaded?

Google led me here: x11 - synclient does not find synaptics properties despite Synaptics Touchpad in xinput list - Unix & Linux Stack Exchange

So in fact the “problem” is that touchpads is nowadays handled by libinput, not by synaptics. This is why xinput still lists the device, but synclient cannot find it.

The touchpad properties can also be controlled using xinput, via xinput list-props and xinput set-prop

Which works! xinput set-prop $device $propID $value, where the property id is given in parentheses in xinput list-props output: libinput Tapping Drag Enabled Default (330): 1

So I (in case gets reset after restart):

xinput set-prop 15 327 1 #enabled tapping
xinput set-prop 15 312 0 1 0 # scroll through side of touchpad

Interestingly, xinput set-prop 15 312 1 1 0 didn’t work, apparently I have to choose one. (Same for “click methods”)

Now we pray the xorg/synaptics drivers I installed at the beginning don’t mess up everything after restart ^^ I followed this: How to Activate Two-Finger Scrolling in Ubuntu 18.04 LTS

More advanced settings for libinput

The ArchWiki is excellent as usual. TIL a tap with three fingers is a shortcut for “paste” and you can change/remap that as everything else! Wow.

TODO - play with buttons and three-taps and two-taps and the physical buttons. Also, where does it define that button N is “paste”? And which clipboard are we talking about?

And - I can do it with my usb mouse!

Day 770

Python parameter unpacking

Extremely helpful answer: Revisions to Passing a dictionary to a function as keyword parameters - Stack Overflow

I also really like this approach:

A few extra details that might be helpful to know (questions I had after reading this and went and tested):

  1. The function can have parameters that are not included in the dictionary
  2. You can not override a parameter that is already in the dictionary
  3. The dictionary can not have parameters that aren’t in the function. Examples:

(Connects with my long-forgotten way of ‘after reading something, ask questions, try to find faults, try to find places this isn’t going to work, try to find connections with stuff you already know, try to find contradictions with stuff you already know’ etc., I have to start doing this again)

Make jira use less whitespace

Main culprit is this code, and changing that value to anything makes life better:

.adg3 .issue-container {
	max-width: 1280px;
}

qutebrowser cycle through css / custom css

This line toggles between solarized-everything1 and the above snippet for making jira wide again.

config.bind(',c', 'config-cycle content.user_stylesheets "~/.config/qutebrowser/css/solarized-dark-generic.css" "~/.config/qutebrowser/css/jira.css"')

Sadly no automatic per-website-css possible yet, it seems.

alphapapa/solarized-everything-css: A collection of Solarized user-stylesheets for…everything?


  1.  ↩︎

Day 769

Updated xrealpath to not include newline

echo -n "string" makes echo not add a newline symbol at the end 1. So anything | xargs echo -n | removes that.

Final command is

xrealpath() {
    realpath "$1"
    realpath "$1" | xargs echo -n | xc
}

  1. linux - echo string | xclip -selection clipboard , copies the ‘string’ but also adds a new line to it. how to fix this? - Stack Overflow ↩︎

Day 764

Noisetorch

Had issues with NoiseTorch microphone not working, fixed by changing the microphone and then back. (…) While I’m at it, updated NoiseTorch, and added this snippet to the polkit config to not-enter passwords: I don’t want to enter my password everytime · lawl/NoiseTorch Wiki

sshfs

Still exists and still works!

  • sshfs me@server:/some/folder /my/local/folder -p 12345
  • umount /my/local/folder
    Can be used to permanently mount stuff through fstab

An insecure faster version is: sshfs -o Ciphers=aes128-ctr -o Compression=no me@server:/some/folder /my/local/folder -p 12345

(In my case, most of my lag was from zsh git prompt plugin, removing it made it much faster)

arandr change monitor settings to get it recognized

When a monitor stops working, sometimes it is fixed by deactivating/applying/activating/applying in arandr, or doing any changes to it intead of deactivating it. I’ve been changing its resolution, but to maximally preserve the layout, just inverting it (and back) works too!

Day 763

nomacs for files over ssh

Nomacs is extremely slow when viewing images located on a remote server, any other viewer works for me. The default one is eog / “Eye of Gnome”

Python investigate memory leaks

tracemalloc is part of the python standard library!

This snippet from the docs1 has everything:

import linecache
import os
import tracemalloc

def display_top(snapshot, key_type='lineno', limit=10):
    snapshot = snapshot.filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "<unknown>"),
    ))
    top_stats = snapshot.statistics(key_type)

    print("Top %s lines" % limit)
    for index, stat in enumerate(top_stats[:limit], 1):
        frame = stat.traceback[0]
        print("#%s: %s:%s: %.1f KiB"
              % (index, frame.filename, frame.lineno, stat.size / 1024))
        line = linecache.getline(frame.filename, frame.lineno).strip()
        if line:
            print('    %s' % line)

    other = top_stats[limit:]
    if other:
        size = sum(stat.size for stat in other)
        print("%s other: %.1f KiB" % (len(other), size / 1024))
    total = sum(stat.size for stat in top_stats)
    print("Total allocated size: %.1f KiB" % (total / 1024))

tracemalloc.start()

# ... run your application ...

snapshot = tracemalloc.take_snapshot()
display_top(snapshot)

  1. tracemalloc — Trace memory allocations — Python 3.9.1 documentation ↩︎

Day 762

Intellij idea commit keybinding

Added <Shift+Alt+C> for “commit”, since <Ctrl+K> doesn’t work (and afaik is not used for anything else). (<Ctrl+Shift+C> is still “copy path”)

Day 759

Intellij idea / pycharm global bookmark a line in the file.

<Ctrl-Shift-#> (where ‘#’ is 1-9) adds named bookmarks to lines in the file; <Ctrl-#> to go there. (It’s logical to make it easier to go to a bookmark than to set one, given that the former should happen more often). Complements nicely ideavim’s m# bindings.

These bookmarks are global.

Intellij idea switch to tab numbers + moving tab + plugings + random keybindings

In the description of the plugin GoToTabs: Now it’s supported natively through keymap->other->tabs! Can’t get tab 2 to work, but I couldn’t do this with bookmarks either, something is catching that binding before it gets to intellij?

Also in idea you can map numpad numbers - I could remap them for bookmarks.

TODO make a backup of my keymap.

And - there’s TabNumberIndicator, that adds the Alt+# bindings and shows the tab number in the tab! Exactly what I wanted.

  • Added <Ctrl+,> for moving the tab left though MoveTab plugin.

EDIT - argh, I knew I needed these Alt+# bindings. TODO change them to Ctrl+Alt+… or similar.

copying a python virtualenv

virtualenv-clone is the package, syntax is 1

python -m clonevirtualenv source/ target/

  1. python - How to duplicate virtualenv - Stack Overflow ↩︎

Day 758

Collision detection of boxes / patterns

This is brilliant: collision detection - What is the fastest way to work out 2D bounding box intersection? - Game Development Stack Exchange

return !(r2.left > r1.right
    || r2.right < r1.left
    || r2.top < r1.bottom
    || r2.bottom > r1.top);

The idea is to capture all possible conditions upon which the rectangles will not overlap, and then negate the answer to see if they are overlapped

Originally from here: Rectangle Intersection – Determine if two given rectangles intersect each other or not « Technical Interview Questions

Doing it straight-forwardly would require more conditions.

Surprisingly intuitive and shows once more that when finding the answer is too hard, trying to find the answer to an opposite question might help you out.

python moving virtualenv makes it use the system default python/pip paths

Python venv (virual environment) uses wrong version of Python - Stack Overflow:

As an addition to the accepted answer, be also aware that changing the directory name where your venv is located causes using the default python and pip paths of your system, instead of using the venv one.

This explains so much!

To make an existing virtualenv movable not included in the new venv. :( 1

No easy official way, reinstalling is much easier.

To find out where a certain package is installed, pip list -v.

Basic Slack bot

import os
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

client = WebClient(token=os.environ['SLACK_BOT_TOKEN'])

try:
    response = client.chat_postMessage(channel='vision-trainings', text="Hello world!")
    assert response["message"]["text"] == "Hello world!"
except SlackApiError as e:
    # You will get a SlackApiError if "ok" is False
    assert e.response["ok"] is False
    assert e.response["error"]  # str like 'invalid_auth', 'channel_not_found'
print(f"Got an error: {e.response['error']}")

Intellij idea applying only some changes from commit in another branch

Find that branch in git log, right click on the file(s) you want, “Apply selected changes”. 2 (“Cherry-pick selected changes” according to Help)

matplotlib add colorbar

fig = plt.figure(figsize=(20, 15))
ax = plt.subplot(132)

#plt.gcf().tight_layout(rect=[0, 0, 1, 0.90])
plt.gcf().tight_layout()

fig.subplots_adjust(right=0.9)
cbar_ax = fig.add_axes([0.92, 0.10, 0.04, 0.8])
if heatmap is not None:
	fig.colorbar(heatmap, cax=cbar_ax)

Confluence page info

Shows incoming and outgoing links, useful to look for other places with similar info.


  1. python - Can I move a virtualenv? - Stack Overflow ↩︎

  2. Apply changes from one Git branch to another—IntelliJ IDEA ↩︎

Day 757

Pycharm / Intellij idea debugging

  • If I highlight/select code before opening the window with <Alt-F8> that code is automatically written there!
  • I should use <Shift+Alt+9>/“Run to cursor” more often
  • I should remember that “scroll to end” exists and should be usually on

Different OCR engines comparison

The Battle of the OCR Engines - Tesseract vs Google Vision | Blog | Fuzzy Labs - really nice! Compares three modes of Tesseract and two Google Vision. TODO add to /f/

timewarrior input time

Timewarrior accepts time the way I usually write it in my notes! timew track 1520 - 1600 stuff just worked!

Day 756

Design / pytorch / ux

I find the “Install pytorch” thing on the pytorch website really nice. You click things and it gives you a command.

CLI program guidelines, to read

Command Line Interface Guidelines - thank you AA “An open-source guide to help you write better command-line programs, taking traditional UNIX principles and updating them for the modern day.”

Day 755

German

New strategy - use only German, look up any grammar stuff I have to, and add the things I have to look up to anki. (Just realized I’m googling whether it’s “dir passt” or “dich passt”, it’s 10/10 an use-case flashcards).

Google colab

.. is really awesome! I should spend some time getting to know it. Example: https://colab.research.google.com/drive/1lzjbBQsF4X2C2WZhxBJz0wFEQor7F-fv?usp=sharing#scrollTo=kbHSUnMRJNTv

Day 752

ssh via public key permissions

Broke log-in to an external server I have access to by attempting to use ssh-copy-id me@server, after which it still wanted my password but once inputted correctly didn’t start the shell. (Though showed the motd).

Day 750

English / Slack

Unfurl | Definition of Unfurl by Merriam-Webster - “expand, extend, fan (out), flare (out), open, outspread, outstretch, spread (out), stretch (out), unfold”

Fastmail calendar

Things I love so far:

  • Can move/change single recurring events without issues, asks whether to do it for one or all of them only when I use the “Edit” button! Things I miss:
  • Ability to “copy” an event in another calendar. Though I consider the need to do this an antipattern, and maybe I’ll find a workflow where I don’t need to do this often.

German / Deutsch

das Teufelszeug - appalling/hellish/infernal stuff (heard at work)

python console vim editing mode!

I so missed this. Adding to ~/.inputrc this line:

set editing-mode vi

makes all readline programs use vi mode, including Python interactive console. Wow.

Alternatively, this apparently works too when typed into python console:

import readline
readline.parse_and_bind("set editing-mode vi")

1

Athame (readline replacement with complete vim support)

ardagnir/athame: Full vim for your shell (bash, zsh, gdb, python, etc)

One can install it in place of the usual readline locally or globally.

Installed for zsh, now I can use ci( bindings again!


  1. Standard python interpreter has a vi command mode? - Stack Overflow ↩︎

Day 749

.vimrc conversion saga

In [Day732]({{site.baseurl}}{% link _posts/2021-01-02-day732.markdown %}), I changed my ./vimrc to utf8 from latin-1, to be able to use the “” symbol to mark trailing spaces.

Well, it broke the vim macros for the link wiki (from [Day 450]({{site.baseurl}}{% link _posts/2020-06-23-day540.markdown %})) :( I had the latin version of the .vimrc backed up, falling back to it for now.

I need to think of a way to save these macros better, because even copypasting them to this dtb didn’t work and I had to do text encoding magic. I think this is exactly the time one should use a proper scripting language like Python, and write another small qutebrowser script that changes the contents of the filled textarea.

link links to pages, post_url links directly to posts inside _posts.

Link to pages:

{%raw%}
{% link _collection/document-name.md %}
{{ site.baseurl }}{% link _collection/document-name.md %}
{{ site.baseurl }}{% link _posts/2019-03-06-post-title.md %}
{{ site.baseurl }}{% link services/index.html %}
{{ site.baseurl }}{% link /assets/documents/pal-codes.pdf %}
{%endraw%}

Links to posts:

{%raw%}
{% post_url 2019-03-06-post-title.md %}
{{ site.baseurl }}{% post_url 2019-03-06-post-title.md %}
{{ site.baseurl }}{% post_url /folder/2019-03-06-post-title.md %}
{%endraw%}

Copied directly from this excellent page, I never found this explained in such a readable way: How to create internal links in Jekyll | Web Island Blog

TODO Jekyll / dtb / meta

Write a small script that allows me to easily link to days just by their day number.

Jekyll changed post permalinks

Before URI contained the date and was hard to link to. Now I changed this in _config.yml:

permalink: :title:output_ext

Links are now like this: https://www.pchr8.net/d/dtb/day749.html

Python representing infinity

float('inf') works for floats, but there’s no way to do it with ints. math.inf is also a float. 1

vim interrupt operation via <Ctrl-C>

Made a typo, vim attempted to indent 20k lines (and started counting “xx lines to indent…”, intuitively pressed <Ctrl-C>, it successfully interrupted the operation!

https://scrolller.com/


  1. Represent infinity as an integer in Python 2.7 - Stack Overflow ↩︎

Day 748

matplotlib reverse colormaps

Every colomap has a reversed version named *_r (such as gray_r)! 1

Papers - NLP - Chargrid

[1809.08799] Chargrid: Towards Understanding 2D Documents


  1. matplotlib.pyplot — Matplotlib 3.3.3 documentation ↩︎

Day 747

Fastmail shortcuts

Keyboard shortcuts | Fastmail

Qutebrowser passthrough

Simplified bindings for passthrogh, added last line to ~/.config/qutebrowser/config.py

config.unbind('<Shift-Escape>', mode='passthrough')
config.bind('<Ctrl-Shift-+>', 'leave-mode', mode='passthrough')
config.bind('<Shift-I>', 'enter-mode passthrough')

Would allow me to use websites' own shortcuts more often.

Day 744

python serialization using dill

dill is like pickle, but serializes more stuff!

python pycharm unittest

Yet another way one can get the “no tests to run” error - if a test is not called test_..., it won’t be ran.

Day 742

i3 sticky window / pin window

It’s easy to do a sticky window in i3!

Added to ~/.config/i3/config:

# Sticky window
bindsym $ms+p sticky toggle

Seaborn catplot titles (plotting, pandas, visualization)

Seaborn anonying facet title · Douglas C. Wu:

sns.catplot(x="target",y="score",hue='score-type',data=d,kind='bar',col='bundle',col_wrap=2,sharex=False,sharey=False).set_titles(col_template='{col_name}')

The set_titles(col_template='{col_name}') removes the usual “x=y” title in each of the sub-plots!

Day 741

qutebrowser crashing

Yet another time qtbrowser started crashing, yet another time fixed it by removing state and sessions from ~/.local/share/qutebrowser/. I blame me messing with qt versions last week.

ag

Somehow magically I don’t have to escape anything in the regexes when using it!

ag "(VISION_|short)" *

passing empty parameters to python argparse / cli?

python - Passing empty string to argparse - Stack Overflow:

python test.py --mode=

I’ve been using args a la -w is, but -w=is also works, and therefore python3 myprogram.py -w -another=opt is perfectly valid! Python parses it as empty string (that casts to False).

fc linux meaning

TIL fc stands for “fix command”!

vim s/ replacing stuff

Discovered that if you just want to remove something, %s/from works (without the second // part at all)

Day 738

pycharm optimize imports

Auto import—PyCharm

python argparse

Seems the best current default way to do cli options! Docs tutorial is as accessible as usual: Argparse Tutorial — Python 3.9.1 documentation

parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) # show default args in help
parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter) # allow newlines in help text
parser.add_argument("-lp", "--localpath", help="Local path. \n %(default)s", default=local_path) # add default text in help text manually

Python shadowing modules

When creating argparse.py, don’t be surprised if you can’t use argparse from import argparse. 1

Python to read, TODO

Nice article: PyFormat: Using % and .format() for great good!

ag as grep alternative for code

I should make an effort to learn it and use it. ag -G "component.*yaml" regex - searches for regex inside all files whose path matches the regex after -G

ag --python "myregex" automatically looks for it in all python files, and really nicely outputs matches!

vim delete lines not containing a pattern

g!/pattern/d, as opposed to the usual g without exclamation mark.

Using less to copy cli stuff with weird linebreaks

If command returns output with newline breaks that are ignored when copypasting directly, using command | less seems to make it work - I can copypaste from there without problems.


  1. argparse module not working in Python - Stack Overflow ↩︎

Day 737

Change volume of bluetooth headphones via cli / pactl

I wasn’t able to do it the usual amixer way, because:

You are running Pulseaudio, which uses ALSA to drive soundcards, but which connects to Bluetooth speakers without involving ALSA. When you set ALSA volumes with amixer, Pulseaudio notices and corrects the source/sink volumes[…] 1

Command to do it directly through pulseaudio is: pactl set-sink-volume name_of_bluetooth_speaker +3%

Added this to ~/.config/i3/config:

bindsym Mod1+r exec  pactl set-sink-volume bluez_sink.60_AB_D2_43_E9_C5.a2dp_sink +5%
bindsym Mod1+c exec  pactl set-sink-volume bluez_sink.60_AB_D2_43_E9_C5.a2dp_sink -5%

Nomacs picture viewer remove animations + frameless

  • Changed transition time to 0 in Settings -> Display -> Slideshow
  • <F10> leaves only the current picture (‘frameless’), a la scrot; Though in this mode drag-n-drop doesn’t work!

zsh text colors list

Found this when autocompleting something else:

(12:36:26/10136)~/ $ which spectrum_ls
spectrum_ls () {
	for code in {000..255}
	do
		print -P -- "$code: %{$FG[$code]%}$ZSH_SPECTRUM_TEXT%{$reset_color%}"
	done
}

Returns 255 lines with 255 colors, they look neat:

Colors

To read - matplotlib

TODO: The Many Ways To Call Axes In Matplotlib | by Jun | Towards Data Science And in general


  1. linux mint - Change volume on bluetooth speaker with amixer - Unix & Linux Stack Exchange ↩︎

Day 736

Deutsch

das wasserzeichen - Watermark! (Heard at work) die dringlichkeit - urgency. “Besondere Dringlichkeit”. Verschiedene Dringlichkeiten. (heard at work)

Bluetooth / Linux

blueman is a nice semi-gui suite for everything. bluetoothctl is an interactive cli.

Linux - remove noise from microphone with Noisetorch

lawl/NoiseTorch: Real-time microphone noise suppression on Linux. - creates virtual devices that are the same as inpucts, but filter the noise. Works really well for me! (Single binary). Works also for filtering voice in outputs! Listening to songs through it is weird.

taskwarrior zsh sprint env variable

Changed date format from %+V to just %V, which gives a sprint like 01 instead of 1 (which in turn removes the need for sprint.is:1 filtering in taskwarrior, now sprint:01 is a unique identifier)

~/.zshrc:

export SPRINT=$(date +%V)

Day 735

matplotlib pyplot make certain color transparent

For this, a subset has to become bad values, and a cmap has to set what to do with them.

my_cmap = copy.copy(plt.cm.get_cmap('gray')) # get a copy of the gray color map
my_cmap.set_bad(alpha=0) # set how the colormap handles 'bad' values
plt.imshow(thing, cmap=my_cmap)

1

As for bad values, I wrote a function similar to this to make them arbitrary:

def get_bad_values(matr, value=0):
	new_matr = matr.astype(np.float)
	new_matr[new_matr == value] = np.nan
	return new_matr

Note that np.nan can only replace a np.float, never an int!


  1. Making image white space transparent, overlay onto imshow() - CrazyGeeks ↩︎

Day 734

Updated i3 config for toggling between modes

Made everything simpler, based on what I usually really need:

bindsym $ms+s layout toggle tabbed stacking
bindsym $ms+Shift+s layout toggle split

TODO - something for “focus tab N in currently focused container”, a la what I have in qutebrowser/intellij.

Yearly dtb ritual of updating year

.. TODO - fix this, finally. +DAY=$(((365)*2+10#$(date +%j)))

ideavim splitters

Added this to ~/.ideavimrc for moving between splits

map <leader>h :action PrevSplitter<CR>
map <leader>l :action NextSplitter<CR>
map <leader>o :action MoveEditorToOppositeTabGroup<CR>

Day 733

record terminal on linux with script

The script utility exists, and is installed by default on at least two systems I have access to. Works really well for interactive sessions!

script --timing=time.txt script.log
scriptreplay --timing=time.txt script.log

Seems to work when ran through screen, even when the screen is detached!

How to Record and Replay Linux Terminal Sessions using ‘script’ and ‘scriptreplay’ Commands

output terminal live on another screen

This is really cool: command line - How to have a terminal mirrored onto a second screen in a two-monitor setup? - Ask Ubuntu

script -f /tmp/lecture1.scrpt
tail -F /tmp/lecture1.scrpt

-f is for “Flush output after each write.” (as opposed to “write everything to the file when script is terminated”)

Day 732

Markdown newline inside quote

Couldn’t understand why there are newlinen in my yearly review blog post from last year. So - in markdown, two spaces and then a line break create a line break.

So, like this:
One
two

Three
Four
Fine, no spaces Six, no spaces

Highlight to see spaces:

So, like this:  
One  
*two*  

> Three  
> Four  
> Fine, no spaces
> Six, no spaces

vim show trailing whitespaces

In connection to the above, yes. Updated ~/.vimrc with the following:

set listchars=tab:\:\ 
set listchars+=trail:◦

Looks like this:
screenshot

vim CONVERSION ERROR - convert file to different encoding / save with other encoding.

For the above had to convert my ~/.vimrc to utf-8, not the default latin-1:
:w ++enc=utf-8

vim insert utf-8 characters

i3 keybinding to make a screenshot and put it into jekyll assets directory

This makes a screenshot as usual, opens it, opens the jekyll dtb assets folder, and puts the screenhsot name in the primary clipboard. I look at the screenshot, if I like it - I drag it directly to the folder, then use the vim/jekyll binding to insert it in the markdown.

bindsym Mod3+Shift+s --release exec scrot -s -e 'mv $f ~/s/screenshots && nomacs ~/s/screenshots/$f & echo -n $f | xclip -selection c && thunar ~/o/dtb/assets/pics/screenshots/'

echo -n is echo without newline (otherwise it gets fed to xc with newline appended). Added to ~/.config/i3/config.

Feels incredibly ugly and unstable but works for me I guess. Ideally it’s long enough to be replaced with a bash script, but not sure it’s worth it. But if I end up doing more of these, I’ll create a one custom big parametrized bash script that I’ll call like ./big-script.sh screenshot.

vim jekyll binding to insert screenshot picture

map <leader>p i![](/assets/pics/screenshots/<esc>pa)<esc>0lli in ~/.vimrc

Inserts a picture with filename from primary selection, then goes back to the description. Used with new i3 screenshot keybinding from above. a in vim is “insert from next character”, so like A but with words.

I really do need to use a/e etc in vim more often.

camel / snake / kebab notation, note to self.

I seem to use more of-this-notation lately, instead of this_notation. Formalize this, not just for consistency, but to use this to my advantage - vim and company see these-words as separate, and this_word as one.

bash echo without newline at the end

echo -n doesn’t add a newline. Especially useful combined with xclip.

Day 730

Haiku

WKD - Matsuo Basho Archives: - Timeline -:

1662 or 1663 寛文二年
His first known hokku at age 19:

春や来し年や行きけん小晦日
haru ya koshi toshi ya yukiken kotsugomori

has spring come
or has the year gone?
second-to-last-day
Tr. Barnhill

what is spring that came
or was it the year that went?
the Second Last Day
Tr. Ueda

Ist das Frühjahr gekommen
oder das Jahr vergangen?
Der vorletzte Tag.
Tr. Udo Wenzel

The Ukrainian translation seems imprecise, but still remains my favourite: Аніяких думок не лишилось в моїй голові наприкінці року!

Чи вже про весну, чи про минулий рік думати? Передостанній день року.

Переклад Геннадія Туркова

Bible

Послание к Римлянам 13:4 – Рим 13:4:

ибо начальник есть Божий слуга, тебе на добро. Если же делаешь зло, бойся, ибо он не напрасно носит меч: он Божий слуга, отмститель в наказание делающему злое.

Послание к Римлянам 13:4 – Рим 13:4: https://bible.by/verse/52/13/4/

Day 728

Taskwarrior / zsh

Updated zsh alias to include non-work tasks tagged +A or +O from current sprint:

s () {task s \(project:w or \(sprint:$(date +%-V) \(+A or +O\)\) \) "$*"}

or has to be lowercase, brackets in taskwarrior’s filtering have to be escaped.

Google sheets linking between spreadsheets

Use a formula like this:

=IMPORTRANGE("https://docs.google.com/spreadsheets/d/1xrGsOD-yXuORqd8cFg21XOo3ZIw9QbSiNDcnSEatlPM/edit#gid=0", "Sheet1!A:A") 1 For me it was ; as separator; may need to confirm connecting sheets.


  1. How to Link Spreadsheets & Share Data in Google Sheets ↩︎

Day 722

Taskwarrior

Changed colour of +F tasks to a better shade of gray in .taskrc

# color.tag.F=color239
color.tag.F=color244

Show all available terminal colors

curl -s https://gist.githubusercontent.com/HaleTom/89ffe32783f89f403bba96bd7bcd1263/raw/ | bash

(yay for curl | bash!)

German

  • Slashes
  • der Schrägstrich - forward slash (/)
  • umgekehrter Schrägstrich - backslash (\)

TODO - all German IT words in one place, and add them to Anki

Intellij idea structural search and replace

Structural search and replace examples—IntelliJ IDEA is neat! Templates are like this:

$Instance$.$MethodCall$($Parameter$)

Note the “target”, usually it’s set to a complete match, but by call and by params exists too.

I had to disable ideavim, because it was doing some weird formatting by itself with brakets and indents that broke checkstyle.

Filters are also important - $params$ will catch only what.function("one"), but won’t what.function(one, two) - had to change count to be [1, +inf]. (1, not 0, because I needed to avoid replacing what.function().)

“Expression cannot be replaced by non-expression” - if you add a ; at the end of the template, it’s an expression now, and the replacement also needs a ; now.

Java “do nothing” instruction (a la pass)

; works! 1 A bigger one that makes checkstyle happy is assert true;

Random / cooking


  1. python - Java do nothing - Stack Overflow ↩︎

Day 721

Deutsch

  • vorkauen -> pre-chew (or “explain with crayons” metaphorically)
  • die Achillesferse - Achilles' heel

Day 718

Google Meet

“Pinning” a screen makes it only big screen you see.

Day 717

German

Day 716

grep escaped quotes without quotes

It’s possible not to use quotes if have just escaped stuff.

grep "\"Thinker" == grep \"Thinker

Til - decompression bomb

After finding a weird error in PIL, found this: Decompression bomb protection · Issue #515 · python-pillow/Pillow

Zip-bombs live on and prosper it seems :)

Day 715

ncdu / ncurses interface to list big directories

ncdu is a little neat program that first scans all the subdirectories of the current one and returns a nice visual navigable list.

d to delete selected folder, ? for help.

zip recursively excluding folder or some files

zip -r result-small.zip ./ -x 'unscoped/*'

-x is a regex of files to exclude. 1

German

dankbare aufgabe - “rewarding task”

Diff two folders

diff -rq folder1 folder2 diffs contents of files inside the two folders.

Daff diff epsilon

daff diff can accept a -d parameter giving an epsilon, changes of numbers smaller than eps won’t be shown in the diff!

screen

screen -r seems to accept not just the full name of the screen to attach, but even just the first letters!


  1. How do I zip up a folder but exclude the .git subfolder - Ask Ubuntu ↩︎

Day 711

qutebrowser

Started to crash suddenly. git pull-ing didn’t help, but it and recreating the virtualenv using scripts/mkenv.py did.

Day 708

German

der werkzeugkasten - toolbox/kit

vim foldignore with foldmethod tab

This makes empty newlines be part of the fold: 1

" Ignore empty lines when using tabfold
:set foldignore=#<CR>

jq sorting

.. exists and works. 2

.one.two|=sort_by('.whatever')|.next.thing


  1. Incorrect behavior on blank lines with foldmethod=indent · Issue #3214 · vim/vim ↩︎

  2. git - How to sort a json file by keys and values of those keys in jq - Stack Overflow ↩︎

Day 704

maven version ranges

(Also used in input of some other internal tools) They are: 1

Range	Meaning
1.0	x >= 1.0 * The default Maven meaning for 1.0 is everything (,) but with 1.0 recommended. Obviously this doesn't work for enforcing versions here, so it has been redefined as a minimum version.
(,1.0]	x <= 1.0
(,1.0)	x < 1.0
[1.0]	x == 1.0
[1.0,)	x >= 1.0
(1.0,)	x > 1.0
(1.0,2.0)	1.0 < x < 2.0
[1.0,2.0]	1.0 <= x <= 2.0
(,1.0],[1.2,)	x <= 1.0 or x >= 1.2. Multiple sets are comma-separated
(,1.1),(1.1,)	x != 1.1

  1. Apache Maven Enforcer Built-In Rules – Version Range Specification ↩︎

Day 702

Python ast literal_eval

literal_eval - ast - Python documentation - Kite

Safely evaluates Python expressions, nice to use for input.

may only consist of the following Python literal structures: strings, numbers, tuples, lists, dicts, booleans, and None.

jq colours

To pipe output with jq to for example less preserving the colours, use -C. (Simialar to --color=always etc.)

jq . $filename -C | less

Day 697

Zsh prompt change based on dev/prod

The script that sets dev/prod sets a variable to 1 if using dev. Based on that variable I get (or not) a red (B) in my zsh prompt.

Did these changes to the theme clean2.zsh-theme:

PROMPT='%{$fg[green]%}(%*/%!)$ZSH_USING_DEV%{$fg[$NCOLOR]%}%B%b%{$reset_color%}%{$fg[blue]%}%B%5c/%b%{$reset_color%} $(git_prompt_info)%(!.#.$) '

ZSH_USING_DEV=""
if [ "$USING_DEV" = "1" ] 
	then
		ZSH_USING_DEV="%{$fg_bold[red]%}[B]%{$fg[$NCOLOR]%}"
	fi

and in ~/.zsrhc

alias uc='. ~/s/setenv.sh p'
alias ud='. ~/s/setenv.sh d'

zsh prompt export variables from a script

Using the usual ./ way doesn’t work!

Note the . ~/s.. in the script above. Running it as ~/what.sh will create a new shell, export the new values there, and close it. Starting with a . will make it run the script without starting a new shell. 1


  1. shell script - How to export an environment variable from .sh file in zsh? - Unix & Linux Stack Exchange ↩︎

Day 696

Confluence tabs + publishing

“Publish without notifying watchers” exits.

If I paste something from vim that has tabs in it in a Code block, in the Edit window it will look fine, but won’t at the end. Editing again will make it again look fine in the edit window. Moral: use spaces.

vim tabs

If you copypaste things from it to code blocks in jira/confluence it will start being weird. You can do expandtabs/retab to convert it to something that looks just like this with spaces when saved. Though feels suboptimal.

Day 695

Multiple monitors / productivity idea

If I have a laptop and two external monitors, put the ‘distracting’ things on the laptop monitor and close the laptop. Open it when I’m officially doing a pause.

My aliases for grep and history

I shouldn’t forget that I have g aliased to grep, along with h to history | grep. Just tried this and it works:

h vim | g http

zsh expand all aliases; zsh keybindings; zsh show all files in directory

Added this 1 to ./zshrc:

expand-aliases() {
  unset 'functions[_expand-aliases]'
  functions[_expand-aliases]=$BUFFER
  (($+functions[_expand-aliases])) &&
    BUFFER=${functions[_expand-aliases]#$'\t'} &&
    CURSOR=$#BUFFER
}

zle -N expand-aliases
bindkey '^E' expand-aliases

^E is <C-e>. Gets run anytime I use it, without connection to the written text. Neat.

Also found this in ./.zshrc:

# Usage:
#   In the middle of the command line:
#     (command being typed)<TAB>(resume typing)
#
#   At the beginning of the command line:
#     <SPACE><TAB>
#     <SPACE><SPACE><TAB>
#
# Notes:
#   This does not affect other completions
#   If you want 'cd ' or './' to be prepended, write in your .zshrc 'export TAB_LIST_FILES_PREFIX'
#   I recommend to complement this with push-line-or edit (bindkey '^q' push-line-or-edit)
function tab_list_files
{
  if [[ $#BUFFER == 0 ]]; then
    BUFFER="ls "
    CURSOR=3
    zle list-choices
    zle backward-kill-word
  elif [[ $BUFFER =~ ^[[:space:]][[:space:]].*$ ]]; then
    BUFFER="./"
    CURSOR=2
    zle list-choices
    [ -z ${TAB_LIST_FILES_PREFIX+x} ] && { BUFFER="  "; CURSOR=2; }
  elif [[ $BUFFER =~ ^[[:space:]]*$ ]]; then
    BUFFER="cd "
    CURSOR=3
    zle list-choices
    [ -z ${TAB_LIST_FILES_PREFIX+x} ] && { BUFFER=" "; CURSOR=1; }
  else
    BUFFER_=$BUFFER
    CURSOR_=$CURSOR
    zle expand-or-complete || zle expand-or-complete || {
      BUFFER="ls "
      CURSOR=3
      zle list-choices
      BUFFER=$BUFFER_
      CURSOR=$CURSOR_
    }
  fi
}

zle -N tab_list_files
bindkey '^I' tab_list_files

<C-i> gives a list of files in the directory, and space-space-tab at the beginning of the line too. <C-q> (push-line-or-edit). More about it here: TIL: save half-typed commands in bash and zsh « Serge Gebhardt (sgeb.io) TL;DR remove command currently being edited and paste it at the next Return.


  1. alias - Resolve all aliases in a zsh command line - Unix & Linux Stack Exchange ↩︎

Day 694

German / English / slang

Seen in the wild at work: ASAPST - like ASAP, but even more urgent. 1

Linux multiple cameras/webcams

Just found this hack: if the program you want to use doesn’t pick the right camera and you can’t control that through settings, open another program that will use the wrong camera - the first program will pick the first free camera, the one you want.


  1. Businesstalk: Neue Begriffe unter Managern. – Leadion ↩︎

Day 693

Jira formatting

Didn’t know that underline is marked +like this+. Why can’t we just agree on a flavour of markdown :(

vim

Changed my main dtb log file from using spaces to using tabs. 1

:set tabstop=2      " To match the sample file
:set noexpandtab    " Use tabs, not spaces
:%retab!            " Retabulate the whole file

Added set listchars=tab:\:\ to vimrc. NB space at the end.


  1. How can I convert spaces to tabs in Vim or Linux? - Stack Overflow ↩︎

Day 689

vim open multiple files via stdin

When trying to do this: find | grep \/model | grep descr | xargs vim -p it opens all files in different tabs as wanted, but breaks the terminal afterwards (need to reset it).

intellij idea tests “No tests found matching Method”

Sometimes I see it and randomly restart and somehow it goes away, today it didn’t.

The usual ‘Invalidate Caches & Restart’ didn’t fix it for me. BUT I had forgotten to annotate it as @Test.

Other ideas about this from StackOverflow: 1

  • run the entire suite, not just that one test
  • If you rename the test it may use the last working running configuration that will fail

Jira markup - indented bullet points / subpoints

Uses asterisks, not indentation.

* I am a bullet point
** I am related to the first one

  1. maven - java.lang.Exception: No tests found matching Method using Intellij IDEA - Stack Overflow ↩︎

Day 687

Intellij idea ‘for’

Didn’t notice it before, but Idea shows the beginning condition of the loop when it’s outside the screen and cursor is on it’s closing braket.

TODO vim

Write something that: a) pastes multiline things automatically at the correct indentation b) copies URIs alone, without leading tabs/spaces. As a bonus - copies only the URI at a certain line without anything else. (I believe I can use kitty for this too, need to look into hints again)

Day 686

screen attach a screen that is ‘attached’ after connection drop

The magic formula I used in the irssi days for all cases works here too: screen -raAd $n

screen scrollback

To change it during runtime, <C-A>:scrollback $number. Otherwise ~/screenrc:

defscrollback 1000000

nextcloud

You can drag-n-drop files from the OS to a folder in nextcloud web gui, and it will automatically upload the file directly there (and inherit all permissions)

Day 683

Linux disable and reenable laptop keyboard

xinput -list

Note the id of “AT Translated Set 2 keyboard” (14 in my case), it’s the laptop keyboard, and of the one labeled “master keyboard” (3 in my case). xinput float $id and xinput float $id $id-master. 1

xinput float 14         #disables laptop keyboard
xinput reattach 14 3    #enables it back

ssh keys (id_rsa, id_rsa.pub)

Errors like:

Permissions 0644 for '/home/me/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.

can be fixed by setting the permissions to 400.

Java complaining about --add_opens + $JAVACMD

For some reason the script took java8 instead of the java11 returned by which java, through $JAVACMD. Temporarily fixed by doing JAVACMD="/usr/bin/java" ./the_script.sh.


  1. How to Disable Laptop Keyboard in Ubuntu or Windows? – TechMagics ↩︎

Day 682

linux screen

If you leave screen in copy mode and detach, it’ll be in copy mode when you attach it back and give the impression that no new output happened from the time you detached it.

Day 681

German ‘alle’ deklination, singular/plural

Das Indefinitpronomen all- - all- -

  • Im Singular bezieht sich aller, alle, alles auf die Gesamtheit unzählbarer Nomen.
    • Wahrscheinlich Plural = countable?
  • Alles steht sehr oft allein und hat eine ganz allgemeine Bedeutung.
    • Alles ist gut.

Day 680

German

Markdown subscript/superscript

Don’t forget that I can use sub/sup HTML tags in markdown!

Day 675

qutebrowser new profile and basedir

Created a new basedir, where I symlinked the config.py from the original one: 11399 5.11.2020 09:54 ln -s ~/.config/qutebrowser/config.py ../profile2/config/config.py

To run with it, python3 qutebrowser.py -B ../profile2 (-B is the same as --basedir)

My use case would be to run a separate instance with tabs on top instead of in the left, for jupyter and similar. I can’t set tab position per window in the same profile.

zshrc aliases for output and copy

As I seem to do that often:

xrealpath() {
    realpath "$1"
    realpath "$1" | xc
}

xpwd() {
    pwd
    pwd | xc
}

Both outputs to screen and saves to buffer. xc is still this:

 alias xp='xclip -selection clipboard o'

pandas nullable integer data type

Nullable integer data type — pandas 1.1.4 documentation Usual int cannot contain NaN values, and things like df[0].astype(int) fail in that case.

To use nullable ints: df[0].astype('Int64'). (Note - it’s a string. int works both ways.)

pandas styling

Pandas can do styling! Had no idea: Styling — pandas 1.1.4 documentation

def _color_original(s):
    if s[5]==-1:
        return ['background-color: lightgrey']*len(s)
    elif s[5]>0.9:
        return ['background-color: #a5a1ff']*len(s)
    elif s[5]>0.8:
        return ['background-color: #bebaff']*len(s)
    elif s[5]>0.7:
        return ['background-color: #d8d6ff']*len(s)
    else:
        #print(s)
        return ['background-color: white']*len(s)

df.style.apply(_color_original, axis=1)

style.applymap() is elementwise, style.apply() is per row/column/table.

Day 674

Slack / Electron apps hard restart

<Ctrl-Shift-R> does a hard-restart that sometimes fixes issues with Electron apps such as Slack. (TS)

pandas invert operator

~ inverts boolean values.

Used like this: dd[~dd.companyName.str.contains("po")] -> “does not contain”

Online buchstabierung Deutsch

Buchstabieren am Telefon auf Deutsch nach DIN 5009 - Ziffern nach DIN 5009 - Satzzeichen und Sonderzeichen

Day 673

Java / German

Sehr schönes deutschsprächiges Buch über Java, dem ich nützen könnte, um mein IT-Wortschatz zu verbessern:

Rheinwerk Computing :: Java ist auch eine Insel - Inhaltsverzeichnis

Intellij Idea commit window

The commit window is (as with conflict resolution) a fully-functioning text editor, including all the usual ideavim bindings! All changes/diffs get automatically updated as they get changed.

Intellij idea copy file

“Copy”-ed files get pasted as the filename with all the extensions. “Copy filename” does only the file name, w/o the last extension.

Intellij environment variables copypaste - check for spaces at the end.

I’ve been bitten by this before at least twice. When copypasting them, for example even from another Intellij running configuration, check for spaces at the end. Not visible at all ever in any of the context one might hope to see them.

Java REPL / shell / cli

Called jshell on my box. Has tab-completion etc. Really nice!

jshell> NavigableSet original = new TreeSet();
   ...> original.add("1");
   ...> original.add("2");
   ...> original.add("3");
   ...>
original ==> []
jshell> original.floor("2")
|  Warning:
|  unchecked call to floor(E) as a member of the raw type java.util.NavigableSet
|  original.floor("2")
|  ^-----------------^
$8 ==> "2"

Java floor/ceil with Pairs and in general, especially with HashMapS and stuff

Think about compareTo() and equals() of the classes if I’m doing something that may need it. Such as using Pairs as keys in a TreeSet and expecting that they will be compared only based on first value.

Java comparing Integers

Java: Integer equals vs. == - Stack Overflow

Using !=/== … works only for Integers -128..127. Otherwise they will likely be different Objects => not equal. And of course, -128..127 is exactly the kind of numbers one might see in tests as opposed to real world situations.

The way to do it is if (!one.equals(two)) {

OWASP Java Gotchas

TODO read this: Java gotchas - OWASP

(Link from AA in the context of comparing integers)

Linux - run command after some time has elapsed

I usually use sleep 5m && foobar, so if I change my mind and ^C the sleep, the next command doesn’t run. – Peter Cordes May 28 ‘16 at 14:07 1

And sleep is installed by default in a lot of places!

sleep 5m && foobar

From the man page:

SUFFIX may be ’s’ for seconds (the default), ’m' for minutes, ‘h’ for hours or ’d' for days. Unlike most implementations that require NUMBER be an integer, here NUMBER may be an arbitrary floating point number.

Duckduckgo (ddg) Wolfram Alpha !bang

!wa now plus 12.5h


  1. shell script - Run command after a certain length of time has elapsed? - Unix & Linux Stack Exchange ↩︎

Day 672

diff side by side

bash - unix diff side-to-side results? - Stack Overflow

diff -y (NB not git diff) does really nice side by side viewing, and still colours output in the columns.

Learning Java

Found Think Java in my old notes and it’s really nice.

Day 668

German

Ctrl+C - “Steuerung-C” - Strg.

Random / interesting / patterns / ML / vision

Intersection over Union (IoU) for object detection - PyImageSearch TODO + add to Fiamma

vim verymagic

This again, but:

  • \v - very magic - everything has special meaning
  • \V - not magic - everything has literal meaning meaning, \ to activate special

Day 667

vim TODO

For my log, it would be interesting to create a keybinding that prepends current time to the beginning of the new created line and stays in insert mode.

Zoom shortcuts

Hot keys and keyboard shortcuts – Zoom Help Center

  • <Alt-A> - mute/unmute microphone.

German resources

Day 665

Confluence creating pages drafts

Confluence saves draft version of the things you write if a page already exists. If the page is not created yet, no draft version is saved.

Pycharm project settings

Project settings are not separate, but part of the general ‘Settings’.

Day 662

Google Sheets / formatting.

Conditional formatting, especially the one that does gradient, is really nice. Butif you enter numbers with “.” as decimal point, while the sheet has “,” as decimal “point”, it will silently fail and color everything white.

Day 661

German / Denglisch

  • Gemerget / Gemergt - no consensus :) 1
  • „mom“ heißt „Moment Mal“ oder „Einen Moment bitte“. 2
  • “Aufruf der Java Platform” = “call” in IT context
  • “gelinde gesagt” = “мягко скажем”; gelinde = mild, soft.

git/bitbucket/intellij pull requests and resolving conflicts

TODO:

TODO grep regex

Lookarounds look useful: Regex Tutorial - Lookahead and Lookbehind Zero-Length Assertions

If TC triggers Sonar, it will provide a link to it once it finishes in the build log! (Same for CheckStyle - if it breaks because of it, going to the error and scrolling up in the log will lead to the problematic file and line.)

tig (git)

TS showed it to me some days ago, installed it, does really nice and colorful visualizations of branches statuses etc etc etc. Keyboard-driven and configurable and basically everything I might want, including easy to remember name.

pandas / pyplot bar plots

This is awesome: Bar Plots in Python using Pandas DataFrames | Shane Lynn


  1. https://twitter.com/blabber/status/582887396460642304?lang=en ↩︎

  2. Was bedeutet „mom“? Abkürzung im Chat und Netz ↩︎

Day 660

Corona Ukraine

Система моніторингу поширення епідемії коронавірусу

Zoom annotations

TL;DR use only one screen and they work.

I have a potential fix for this! I have 2 screens (internal laptop monitor, and external monitor). I noticed that the problem only presents when I’m using “dual monitor” mode and have “gallery view” enabled. The controls also don’t appear if I have “full screen” enabled. They also won’t appear if I have either window on the external monitor.

It works in “dual monitor” mode if I have the controls window set to “speaker view” and not in “full screen” with both windows on the internal (laptop) monitor. I can then move either window to the external monitor. If I’ve accidentally started Annotation mode “wrong”, I have to cancel it and move all windows to the internal monitor before re-starting Annontation mode.

I have my controls back! This is with version 5.2.458699.0906 1

I have version 5.3.47…. (no way to copypaste) and they don’t, but using only one monitor helped. Though I get a black screen underneath annotations when/as I’m writing them.


  1. Annotate controls don’t show up - Zoom for Linux (Pop OS 20.04) : Zoom ↩︎

Day 659

zsh alias pwd | xc

alias pwx='pwd | xc'

I really need to gather them all in one place and make ~/.zshrc cleaner and neater.

bash / python / whatever todo small things


```
cc() python3 -c "from math import *; from statistics import *; print($*);"
alias cc='noglob cc'
```

Added `statistics` to it, now I can do things like `cc mean([2, 4, 9])`. (As a bonus `stdev()` etc., [full list](https://docs.python.org/3/library/statistics.html)). Works for now. `"`, if any, have to be escaped. 


## zsh prompt
Yet another change (theme file `clean2.zsh-theme`):
`PROMPT='%{$fg[green]%}(%*/%!) %{$fg[$NCOLOR]%}%B%b%{$reset_color%}%{$fg[blue]%}%B%5c/%b%{$reset_color%} $(git_prompt_info)%(!.#.$) '`

It transforms to:

`(10:23:09/10712) some/folder/whatever/ $ mycommand`

Removed the user (that I had to edit out often), moved current time from `PROMPT` to the left side (now I can happily copypaste CLI args and output again).


## regex greedy / lazy
[Regex Quantifier Tutorial: Greedy, Lazy, Possessive](https://www.rexegg.com/regex-quantifiers.html) TODO

TL;DR most regex engines are greedy by default. Greedy = 'longest possible match'. Making it lazy sometimes means adding a "?" to the quantifier, such as `.*?`. Not supported everywhere, notably grep does this only with the perl syntax (`grep -P .*?`).

For vim: [regular expression - How to make regex matchers non-greedy? - Vi and Vim Stack Exchange](https://vi.stackexchange.com/questions/196/how-to-make-regex-matchers-non-greedy/13363#13363)

## `info` command instead of `man`
Nice, had no idea this existed. `info diff`, for example.
Found here: [linux - Understanding of diff output - Unix & Linux Stack Exchange](https://unix.stackexchange.com/questions/81998/understanding-of-diff-output).

## Unified diff how to read

`@@ -1,4 +1,5 @@`: 
> - The number after the comma is the length of the chunk (in the old or new file), and
> - the number before the comma is the line number for the beginning of the chunk.
> - The + and - signs refer to adding and deleting lines, respectively.  [^sounce]

[^sounce]: [What do the numbers in the @@ line mean in output of diff? - Unix & Linux Stack Exchange](https://unix.stackexchange.com/questions/252927/what-do-the-numbers-in-the-line-mean-in-output-of-diff?noredirect=1&lq=1)

Gnu diff man page has more: [Detailed Unified (Comparing and Merging Files)](http://www.gnu.org/software/diffutils/manual/html_node/Detailed-Unified.html)

## Git / Jira / Bitbucket branches and connections to tickets
As long as the branch name contains the issue key (ABCD-123), it gets connected - but only if the branch is created _after_ the creation of the ticket. [^jibbranchsource]
> As stated previously in this question, it is not possible. You are screwed if you create the branch first.

But also it seems that mentioning an issue name in the pull request (does this work for commits?) also works:
> Simply renaming the Pull Request in Github to include the issue id XXX-nnn in the name triggered that PR and Branch to be linked to the Jira ticket.  [^jibbranchsource]

[^jibbranchsource]: [Solved: How to link existing JIRA issue to a existing bran...](https://community.atlassian.com/t5/Jira-questions/How-to-link-existing-JIRA-issue-to-a-existing-branch-in/qaq-p/701496)

## Random / i3 / workspaces
I could define a workspace officially, like number 5, for terminals I don't really need but that are open in deep folders I might need later.

## Random / documentation / dtb
Sometimes I miss the ability to drag and drop screenshots to my textfile with descriptions of stuff I did. I can drag and drop screenshots but they are a bit ephemeral. An interesting idea would be create a different keybinding that creates screenshots that get put in a particular folder; I can still drag-and-drop them, but I'll have stricter guarantees that they'll be there when I'll look for them.

## vim plugin MultipleSearch
[MultipleSearch - Highlight multiple searches at the same time, each with a different color. : vim online](https://www.vim.org/scripts/script.php?script_id=479)

```
:Search sth
:Search somethingelse
:SearchReset
```

Highlights them in diff colors, just what I need. `n/N` works only on the last one.

[vim-scripts/MultipleSearch2.vim: Improve Multiple search](https://github.com/vim-scripts/MultipleSearch2.vim) looks like an improved version of the above.

## vim search and substitute
You can use `\1` etc for `%s/..` commands, if the groups where previously defined by you in a usual `/search-for-stuff`! 
```
/\v(https?):\/\/(.{-})\/.*        <-- Search
:%s,,Protocol:\1 - Domain:\2,g    <-- Substitution
```

This is awesome.

Day 658

Jira

Tickets are linked automatically when mentioned like WHAT-1234, but only with spaces around them; WHAT-1234: result, for example, wouldn’t work.

zsh alias for summing up a column of numbers

alias sumup='paste -sd+ | bc'

bash sort version numbers

This is nice! sort -V sorts by “version numbers” => 1, 10, 20, 50, 200, instead of the alphabetic 1 10 20 200 50.

bash zsh history / aliases TODO

I need something like “output to CLI and copy to clipboard” + I need a smaller timestamp and to the left, not right-aligned, so I can more easily copypaste stuff.

Day 656

Quotes / HN

I think part of the problem is people being raised to believe that love, in the biochemical sense, is the most important part of a relationship. Hence when they start feeling similar feelings for others, their relationship can weaken. But a relationship can be about so much more than just the biochemical side: two people, with similar goals and values, working together to build a better life for themselves and their children, and to help each-other achieve their dreams. 1

Random / Interesting

Zersetzung - Wikipedia “(German for “decomposition”) is a psychological warfare technique used by the Ministry for State Security (Stasi) to repress political opponents in East Germany during the 1970s and 1980s.”

Home wifi/ethernet speed comparison

TL;DR Download speed with ethernet is 4x faster, no differences otherwise.

Sa 17 Okt 2020 17:15:44 CEST.

Wifi, 50cm from router:

$ speedtest-cli

Retrieving speedtest.net configuration...
Testing from [redacted]
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by [redacted]: 46.246 ms
Testing download speed................................................................................
Download: 101.10 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.64 Mbit/s

Ethernet from same router:

$ speedtest-cli

Retrieving speedtest.net configuration...
Testing from [redacted]
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by [redacted]: 40.086 ms
Testing download speed................................................................................
Download: 395.90 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.89 Mbit/s

  1. He Married a Sociopath: Me | Hacker News ↩︎

Day 654

Added zsh alias for copying current dir

alias ccd='pwd | xc'; though I kinda feel the task would be solved nicely as keyboard shortcut for kitty. + Reminder to myself that zsh nicely cd-s to it when pasting the folder, without cd.

Also find | grep is now f instead of ff.

German

“Du spricht für uns alle”

English / interesting

grep for looking for tests in different files

Forgetting about this nice use of grep:

grep "whatever" * gives the matches while also displaying the filename!

Day 653

Intellij Idea Ideavim actionlist

And yet another place to find this! :actionlist as command in ideavim.

Ideavim jump to place of last edit

IdeaVIM Help: motion is the last readable docu I can find, from 2006.

gi starts insert mode at the last place insert mode was exited; '^ just moves the cursor there.

Intellij Idea tests not running

Had an issue with “Class not found” when running tests, fixed with “Invalidate caches and restart”

Intellij up/down keys

My custom up/down keys set via xmodmap (<Super-C> etc.) got recognized everywhere in the OS except intellij; Can get configured in intellij keymap.

Day 652

Intellij idea toolbars

“View -> Appearance -> Tool window bars” gives extremely useful tool window bars that include a number, the one I can use in <Alt-N> bindings to make them appear! Same goes for the …tabs to the right with Structure etc.

Intellij building complex stuff and running maven goals

Once N maven goals or whatever are run and N tabs opened, they can be found again in the “Run” tool window bar (<Alt-4>). Particular tabs can be also successfully restarted.

maven skip checkstyle

mvn [target] -Dcheckstyle.skip does the magic without editing any POMs etc. 1

Java Comparators

  • Not bound to -1, 0, +1, any positive/negative number works.

Random / interesting

Nightcore - Wikipedia - A nightcore edit is a cover track that speeds up the pitch and time of its source material by 10–30%.


  1. Disable maven checkstyle - Stack Overflow ↩︎

Day 651

German

in trockenen Tüchern - “cut and dried” = “final”. See in trockenen Tüchern - Wiktionary

Java profiling with visualVM

Day 648

scp can do autocompletion!

Nothing to add, but I think it works if you have access via public key instead of passwords. scp what-ever:/home/whatever/..<TAB> works nicely with zsh even visualizing it like usual.

Also TODO rsync apparently can do the same things as scp, but better, I should give it a chance.

Pandas / matplotlib

It’s possible use panda’s graphs as an ax and do all other modifications to it!

ax = data.plot(y=['tp','fn'], kind='bar',stacked='true')

jupyter / mpl / google docs

You can drag and drop graphs directly from jupyter into a google docs presentation

Day 647

grep / uniq magic

uniq can compare stuff before / after N characters in the string, group together duplicate ones, etc!

  • uniq -w N looks at only the N first characters.
  • uniq -s N don’tt look at the first N characters
  • uniq --all-repeated[=METHOD] prints all duplicated lines and optionally separates them via METHOD. separate is really nice as method.
  • uniq -c prefixes line by number of occurrences of that line

And I guess again - uniq works only on sorted data.

Day 646

Slack formatting

Format your messages | Slack:

  • ~strikethrough~, *bold*, _italic_.
  • > quoting is as usual.
  • Highlight text and paste a link -> create a link.
in:#team-marketing 
in:@sara

to:@anna

before:4/12/2019
after:4/12/2019
on:4/12/2019
during:april

has::white_check_mark:

has:pin
is:saved
has:link

vim folding by tab works again now!

Had to remove this snipped from vimrc:

augroup remember_folds
  autocmd!
  autocmd BufWinLeave *.* mkview
  autocmd BufWinEnter *.* loadview
augroup END

vim folding

Will now try to use almost exclusively folding by tabs. For this, these are going to be helpful: 1

zo		Open one fold under the cursor.  
zc		Close one fold under the cursor.  

zO		Open all folds under the cursor recursively.  
zC		Close all folds under the cursor recursively.  

za		When on a closed fold: open it.  
		When on an open fold: close it 
zA		When on a closed fold: open it recursively.
		When on an open fold: close it recursively 

zv		View cursor line: Open just enough folds to make the line in
		which the cursor is located not folded.

zX		Undo manually opened and closed folds: re-apply 'foldlevel'.
		Also forces recomputing folds, like |zx|.


zm		Fold more: Subtract one from 'foldlevel'.  
zM		Close all folds: set 'foldlevel' to 0.


zr		Reduce folding: Add one to 'foldlevel'.
zR		Open all folds.  This sets 'foldlevel' to highest fold level.


zn		Fold none: reset 'foldenable'.  All folds will be open.
zN		Fold normal: set 'foldenable'.  All folds will be as they
		were before.
zi		Invert 'foldenable'.


MOVING OVER FOLDS 

[z		Move to the start of the current open fold.  
]z		Move to the end of the current open fold.  
zj		Move downwards to the start of the next fold. 
zk		Move upwards to the end of the previous fold.  

TODO and XXX conventions

coding style - Where does the TODO convention come from? - Stack Overflow - XXX as marker for something dangerous. Fixme Comment in c2 wiki for more.

TODO vim highlighting

Very ugly but it works:

fun! SetTodo()
    syn match modo /.*MODO.*$/
    hi link modo Keyword
    syn match nodo /.*NODO.*$/
    hi link nodo javaScript
    syn match todo /\s*TODO.*$/
    hi link todo vbTodo
    syn match done /\s*DONE.*$/
    hi link done cssListProp
    syn match xxx /\s*XXX.*$/
    hi link xxx htmlError
endfu
autocmd filetype markdown :call SetTodo()

  1. Vim documentation: fold ↩︎

Day 645

How to prettyfy json using jq

jq . inputfile.json gives a nice pretty colored indented output.

Wolfram alpha “time between”

https://www.wolframalpha.com/input/?i=time+between+2020-10-02+16%3A55%3A01+and+2020-10-03+20%3A00%3A29

Works also in this shorter format: https://www.wolframalpha.com/input/?i=time+between+2020-10-06+11%3A22%3A50+2020-10-06+14%3A53%3A49 (time between $date1 $date2)

Day 641

screen attaching non-existing named screens creates them

If I do screen -R 20, and there’s no screen named ‘20’, such a screen will be created and attached. Neat.

killing all screens

Quitting all screens from inside screens is the screen command C-a \ 1


  1. linux - How do I kill all screens? - Unix & Linux Stack Exchange ↩︎

Day 640

i3 shortcuts for monitors

Remembered about and updated the shortcuts for focusing the different monitors:

# focus outputs screen
bindsym $ms+a focus output eDP-1
bindsym $ms+o focus output HDMI-2
bindsym $ms+e focus output DP-1-1

Connected to me remembering about the tabbed layout:

# change container layout (stacked, tabbed, toggle split)
bindsym $ms+s layout stacking
bindsym $ms+Shift+t layout tabbed
bindsym $ms+Shift+s layout toggle split

takes less space than my usual stacked, but makes it hard to focus a different screen via left/right - until now.

bash rm multiple folders

For removing multiple folders if it has to be done often, and the folders may or may not be there, esp. with wildcards, current best option I found is not to do rm -rf one/* two/*something* three but rm -rf one/* && rm -rf two/*something* && rm -rf three. If one breaks the other ones will still run.

zsh removing an alias + zsh feature of ‘you sure you want to delete all N files from… ' + rm -r **

  • unalias rm 1
  • The feature can be removed by an option, but this is nice:

I don’t know why but the following works for me - no questions asked [..]

rm -rf **

Works for me too.

Random / phrases

That answers a question, but not this one! :) – Lambart Jul 28 ‘17 at 0:43 2

Intellij Ideavim full list of keybindings + configured movements.

ideavim/package-info.java at master · JetBrains/ideavim

 * |CTRL-B|               {@link com.maddyhome.idea.vim.action.motion.scroll.MotionScrollPageUpAction}
 * |CTRL-D|               {@link com.maddyhome.idea.vim.action.motion.scroll.MotionScrollHalfPageDownAction}
 * |CTRL-F|               {@link com.maddyhome.idea.vim.action.motion.scroll.MotionScrollPageDownAction}
 * |<BS>|                 {@link com.maddyhome.idea.vim.action.motion.leftright.MotionLeftWrapAction}
 * |CTRL-H|               {@link com.maddyhome.idea.vim.action.motion.leftright.MotionLeftWrapAction}

Half-page-down (<Ctrl-D>) doesn’t work in the output of running programs / debuggers, but <Ctrl-F> always does. (And <Page Down> too, ofc.)

ls -l, last modified and files from last year

me@server:/some/folder$ ls -larh ..
total 290M
drwxr-xr-x  3 me users 4.0K Jul  7 16:51 file
drwxr-xr-x  4 me users  12K Jul  7 16:51 file
drwxr-xr-x  4 me users 4.0K Jul  7 16:42 file
drwxr-xr-x  2 me users 4.0K Jul  7 16:32 file
drwxr-xr-x  4 me users 4.0K Jul  7 16:32 file
-rw-r--r--  1 me users    0 Jul  7 16:19 file
-rw-r--r--  1 me users 4.0K Jul  7 16:19 file
-rw-r--r--  1 me users    0 Jul  7 16:31 file
-rw-r--r--  1 me users 1.6K Jul  7 16:32 file
-rw-r--r--  1 me users 144M Oct  1 13:23 file
drwxr-xr-x  7 me users 4.0K Oct  1 13:35 file
-rwxr-xr-x  1 me users 4.9K Oct  2  2019 file
-rw-r--r--  1 me users  46K Oct  4  2019 file
-rw-r--r--  1 me users  45K Oct  2  2019 file
-rw-r--r--  1 me users 146M Oct  2  2019 file
drwxr-xr-x  2 me users 4.0K Oct  2  2019 file
-rwxr-xr-x  1 me users 3.8K Jul  7 16:32 file
drwxrwx--- 13     500 users 4.0K Mar 27  2020 ..
drwxr-xr-x  9 me users 4.0K Sep 30 17:10 .

If files are not from this year, it shows the year instead of the time!

hashing / checksum utility jacksum

Jacksum - a free cross platform checksum utility Is also in the repos as jacksum.

jacksum -a hashingAlgo -x filename

Slack zooming / font size

<Ctrl-plus> and <Ctrl-minus> increase / decrease Slack font size!


  1. Double rm verification in zsh - Unix & Linux Stack Exchange ↩︎

  2. IntelliJ: Viewing diff of all changed files between local and a git commit/branch - Stack Overflow ↩︎

Day 639

head / tail starting from an offset

bash - Linux head/tail with offset - Stack Overflow:

 -n, --lines=K
        output the last K lines, instead of the last 10; 
        or use -n +K to output lines starting with the Kth

You can therefore use ... | tail -n +2 | head -n 3 to get 3 lines starting from line 2.

Simple but effective. tail -n +N does tail starting from linenumber N.

Intellij idea search / find

Search is search, but “find in path” (and some other stuff) is find. I wonder if there’s a meaningful semantic difference in there. Semantic difference between Find and Search? - Stack Overflow has the logical ones, but I can’t connect it to how intellij idea uses them.

Day 638

i3 disable changing workspace via mouse wheel

bar {
    wheel_up_cmd nop
    wheel_down_cmd exec ~/.i3/scripts/custom_wheel_down
}

1 is an example. Disabled it in config, not the i3 statusbars own configs.

Intellij idea debugging step out

“Step out” (<Shift-F8>) skips to the first line executed after returning from the current method.

Intellij idea debugging / run code fragment

<Shift-Return> changes the window from one line to multi and back.

tee vs less with UTF-16 / UTF-16LE encoding + reading them in vim

If a file is UTF-16 encoded, tee works much better than less.

To read UTF-16LE in vim, the command is: :e ++enc=utf-16le

Todo diensttagebuch

In my text version of this, I need a nice way of interlinking parts of the text. Ideally in a way that adding text doesn’t break it, and in a way that it’s still text-safe.


  1. Disable switching workspaces or windows via mouse · Issue #1104 · i3/i3 ↩︎

Day 635

Screenkey

screenkey1 (in the repos too) shows the keys you pressed on the screen. screenkey --show-settings shows a nice GUI before start with all the configs.

Looks like the second answer here: folding - Is there a best practice to fold a vimrc file - Vi and Vim Stack Exchange

Vim foldmodes

This answer has an awesome idea for folds, something very close to what I was looking for in my long notes file: folding - Is there a best practice to fold a vimrc file - Vi and Vim Stack Exchange

In general TODO vim folds - create a nice stable system instead of the ad hoc hell I have now.

Also How to enable and use code folding in Vim? - Unix & Linux Stack Exchange has a nice summary of fold keybindings:

vim keybindings

I forgot I used <leader> for most of the things I’m currently using F12 for!

Work-home keybindings

TODO I guess A shell script that changes monitor configurations, volume, changes some of the zsh aliases to show me the correct taskwarrior todos, etc.

Something like this:

#!/bin/zsh
if [ "$1" == "work" ]
then
    echo "Setting configs to work..."
    export SOMETHNIG="something" ...
    ...
else
    echo "Setting configs to home..."
    export SOMETHNIG="something" ...
    ...
fi
zsh

How do I get all open terminals to use the new env. variables though?


  1. wavexx/screenkey: A screencast tool to display your keys inspired by Screenflick ↩︎

Day 634

Markdown syntaxt highlight in vim and URIs

If you paste an URL containing characters with a meaning in markdown, it breaks syntax highlight till the end of the file. Especially URIs containing an uneven number of underscores. Fix is either make it code, or indent it - it seems formatting is local for indentation, at least for vim.

Parsing json via jq


It supports newlines in filters!

To read

“Data science and the command line” (from json - How to nicely remove empty array in jq - Stack Overflow}

vim / terminal drag and drop

I can drag and drop my screenshots in the terminal and it pastes the absolute location!

uniq works only on sorted output!

Wow. …wow. command line - Uniq won’t remove duplicate - Unix & Linux Stack Exchange

This explains really a lot.

Day 633

Java logical operators shorct-circuiting

The && and || operators “short-circuit”, meaning they don’t evaluate the right-hand side if it isn’t necessary. The & and | operators, when used as logical operators, always evaluate both sides.

There is only one case of short-circuiting for each operator, and they are:

  • false && ... - it is not necessary to know what the right-hand side is because the result can only be false regardless of the value there
  • true || ... - it is not necessary to know what the right-hand side is because the result can only be true regardless of the value there. 1

Quoting quotes and answers with code from Stack Overflow

Clicking on “improve this answer”, even when not logged in, gives access to the source markdown that I can copy here preserving most formatting.

Stand-off markup (NLP)

Stand-off markup (also known as remote markup or stand-off annotation) is the kind of markup that resides in a location different from the location of the data being described by it. It is thus the opposite of inline markup, where data and annotations are intermingled within a single location. 2

The wiki it’s from is also nice: TEIWiki It’s All pages - TEIWiki would be interesting to look through, NB it has a non-intuitive “next page” link there.

taskwarrior TODO

I need either a command or an alias for “create task and start immediately”, along with some nice way to create boilerplate code for tasks

Intellij idea debugging “Drop Frames”

“In case if you want to “go back in time” while debugging you can do it via Drop Frame action. This is a great help if you mistakenly stepped too far. This will not revert the global state of your application but at least will get you back by stack of frames.” 3 (Icon with a red X on top left.)

Java declaring variables inside or outside a loop

Does it make a difference if I declare variables inside or outside a loop in Java? - Software Engineering Stack Exchange - TL;DR it doesn’t, if you are not creating complex objects. The way to learn this is to look at the generated bytecode, using javap.


  1. Java logical operator short-circuiting - Stack Overflow ↩︎

  2. Stand-off markup - TEIWiki ↩︎

  3. Debugger - IntelliJ IDEA - Confluence ↩︎

Day 632

vi mode / readline / inputrc / history

This is freaking awesome:

k: history-search-backward
j: history-search-forward

… in ~/.inputrc allows using j/k for scrolling through history in vim mode. 1 This configures readline that then is used by (at least) bash, zsh, and fish.

ag command

I still have it installed as a standalone command, I should use it. Ag (silver searcher) is better than ack, which is better than grep. ggreer/the_silver_searcher: A code-searching tool similar to ack, but faster.

TODO get proficient with them and learn how exactly they are better for writing code.

German

dict.cc | muskelkrampf | English Dictionary - muscle cramp

Intellij Idea “Find in path”

<Ctrl-Shift-f> for “Find in path” - extremely useful for large codebases; info from Tony.


  1. No script is too simple | Hacker News ↩︎

Day 631

Random / Interesting / Quotes

“The best minds of my generation are thinking about how to make people click ads.” –Jeff Hammerbacher

Found on I no longer build software | Hacker News in the context of Allow docker push' to push multiple/a subset of tags · Issue #267 · docker/cli · GitHub:

@solvaholic: Sorry I missed your comment of many months ago. I no longer build software; I now make furniture out of wood. The hours are long, the pay sucks, and there’s always the opportunity to remove my finger with a table saw, but nobody asks me if I can add an RSS feed to a DBMS, so there’s that :-)

Slack ‘set status’

In ‘set status’ there’s a ‘clear’ option, it controls when the status gets reset, minutes/days.

Emacs tutorials

Uncle Dave - YouTube (No script is too simple | Hacker News) - for later. Notable there is:

Day 627

Intellij idea debugging “run to cursor”

  • “Run to cursor” is <Alt-Shift-9> and the ideavim cursor works!
  • “Show exectution point” <Alt-F10> is also really useful

redshift

Remembered that redshift exists, added it to startup. It’s the program that makes screen redder at night, a la twilight / night mode / etc etc etc.

Day 626

Intellij reopening closed windows such as debugging

Reopening closing windows can be done through View -> Tool windows. It has nice shortcuts that are <Alt-N>, with N=0..9. For example, debug window is <Alt-5>.

Additionally - I should use F7 / ‘step into’ much more.

Intellij idea structure

<Alt-7> opens a nice structure window with info about the long class you’re editing. With nice options like show public/fields/..

Gson / json serialization of objects

java - Save state of object in IntelliJ debug? - Stack Overflow - google/gson is the answer. User guide: gson/UserGuide.md at master · google/gson · GitHub

Gson gson = new Gson();
gson.toJson(1);            // ==> 1
gson.toJson("abcd");       // ==> "abcd"
gson.toJson(new Long(10)); // ==> 10
int[] values = { 1 };
gson.toJson(values);       // ==> [1]

// Deserialization
int one = gson.fromJson("1", int.class);
Integer one = gson.fromJson("1", Integer.class);
Long one = gson.fromJson("1", Long.class);
Boolean false = gson.fromJson("false", Boolean.class);
String str = gson.fromJson("\"abc\"", String.class);
String[] anotherStr = gson.fromJson("[\"abc\"]", String[].class);

Not drag-and-drop for more complex stuff though.

Intellj Idea Exceptions Breakpoints

You can create breakpoints at all exceptions of a certain type, even when they are caught. May lead to discovery that there are a lot of them in the code :)

On this topic - “Run to cursor” is nice

Day 625

English / Interesting

Tohu wa-bohu - Wikipedia “is a Biblical Hebrew phrase found in the Genesis creation narrative (Genesis 1:2) that describes the condition of the earth (eretz) immediately before the creation of light in Genesis 1:3.”

Now let’s play with Jekyll and UTF-8.

וְהָאָ֗רֶץ הָיְתָ֥ה תֹ֙הוּ֙ וָבֹ֔הוּ וְחֹ֖שֶׁךְ עַל־פְּנֵ֣י תְהֹ֑ום וְר֣וּחַ אֱלֹהִ֔ים מְרַחֶ֖פֶת עַל־פְּנֵ֥י הַמָּֽיִם‎

— Genesis 1:2, Westminster Leningrad Codex

Now the earth was formless and empty, darkness was over the surface of the deep, and the spirit of God was hovering over the waters.

— Genesis 1:2, New International Version

Better master file

So, before we had this file:

echo "Concatting all existing days"
cd _posts
echo \
"---
layout: page
title: Master file
permalink: /master/
---
"\
> ../master_file.md

cat * | grep -v "layout: post" | grep -v "categories: \[" | grep -v "title:  \"Day" | grep -v "date:   20" | grep -v "\-\-\-"  >> ../master_file.md

This didn’t really work well for various reasons, now I did this right:

{% raw %}---
layout: default
---

<div class="home">
  {%- if page.title -%}
    <h1 class="page-heading">{{ page.title }}</h1>
  {%- endif -%}

  {{ content }}

      <ul class="post-list">
      {% for post in site.posts %}
      <h1><a href="{{ post.url }}">{{ post.title }}</a></h1>
        <p class="author">
          {% assign date_format = site.minima.date_format | default: "%b %-d, %Y" %}
          <span class="post-meta">{{ post.date | date: date_format }}</span>
        </p>
        <div class="content">
          {{ post.content }}
          <br>
        </div>
      {% endfor %}
    </ul>
</div>{% endraw %}

Master file is at the same URI: Master file | Diensttagebuch

Jekyll raw

The nice tags for raw input, of course, work immediately, no spaces between (%raw%)And the start of the actual content(%endraw%).

Day 624

English / German

dict.cc dictionary :: Die Kacke ist am Dampfen :: German-English translation == shit hitting the fan

Intellij idea running configurations

One can export it to ~/.idea/runConifgurations/*.xml which can be edited with any editor or copied to another project folder.

Intellij idea enable VCS integration

Had this issue when creating a project in a folder that was not equal to the folder with the project files downloaded from git. It did not automatically recognize .git, but I could do it from the VCS menu, something like “choose VCS type”->git etc. Disappeared after I did it.

From Error on openjdk11, cannot find javadoc · Issue #26 · chenjianjx/wsdl2html · GitHub: export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::")

readlink resolves symlinks. Alone it just gives you the file the symlink points to. readlink -f follows each and every symlink to the very end. On my box, the difference is:

me:~/ $ readlink /usr/bin/javac     
/etc/alternatives/javac
me:~/ $ readlink -f /usr/bin/javac  
/usr/lib/jvm/java-11-openjdk-amd64/bin/javac

BUT! The actual JAVA_HOME thing should be solved as a setting in Intellij->Maven, as noted in Day 554 | Diensttagebuch.

Day 623

English

TODO Ankify: spelling of Colleague

German

schnipsel | Übersetzung Englisch-Deutsch - scrap/shred/snipped (noun). Mentioned by Caro in context of Schnitzeljagd – Wikipedia that is an interesting thing by itself.

git coloring

git --color=never means that no color gets added by git, but any existing is preserved. Usually the program piping the output has to be told that colours are still needed despite the fact that it’s not outputting to a terminal.

bash pretty CSV viewer

Pretty CSV viewing on the Command Line - Stefaan Lippens inserts content here this is realy nice! In goes a CSV, out goes a pretty CSV. With all the tabs right etc. Piping output from daff diff to this works wonders.

Added his function to .zshrc:

function pretty_csv {
    column -t -s, -n "$@" | less -F -S -X -K
}

Day 619

vim antipatterns w vs e + vim movements

I should use e much more than w, as that’s what I really usually need. And in general I should take the time to get to know better the tools I’m constantly using. I guess ‘right’ vim is something like ‘right’ typing.

Also, never used them like this, but c of course takes all movements, so changing next two lines or changing from cursor to beginning of the line also works.

Jira collapsible / spoiler block + code syntax highlighting

This 1 is how you do collapsible spoilers in Jira:

{code:bash|title=Header for code|collapse=true}
echo this is some code
{code}

Shows also how to define which syntax highlighting to use.

kitty paste to terminal and copy url hints

Added this to config: map kitty_mod+p>c kitten hints --type path --program @ - copies hinted URL to clipboard.

Default map kitty_mod+p>f kitten hints --type path --program - - pastes the hint to the current terminal window.

This gets rid of so much copypasting oh my God.

Libreoffice calc freeze rows and columns

The setting for freezing headers and stuff so that they remain visible wherever you scroll is in View -> Freeze cells

bash sort by multiple columns

sort -k 13 -k 15 -k 7 -t ";" -d filename - here the multiple -ks are the different sorting keys. -t is separator, and -d is dictionary order. Interestingly, it automatically sorts by everything else too after the column you give it - if you want it to look only at that one column, you do -k 2,2 2


  1. Allow jira {panel} and {code} blocks to be collaps… ↩︎

  2. Trying to sort on two fields, second then first - Unix & Linux Stack Exchange ↩︎

Day 618

Random / Interesting

vim regex separator

I have been using \s as “word separator”, this is wrong. \s in vim is just a whitespace character - so tabs, spaces etc. For what I usually want, \A is any non-alphabetic-character. (\a being an alphabetic one).

vim regex inverse match capturing group + ‘very magic’ mode

/\vcat (dog)@!
/cat \(dog\)\@!

First one is very magic, second is the same without it. @! is a negative match to the preceding capturing group (could have been (\1) too).

\v activates “very magic” mode where everything except letters and underscore has special meaning. This is actually going to be very handy. 1

I’ve actually been using regexes quite a lot lately, have no idea how I lived without them when parsing very big log and text files.

vim paste last searched regex

:s/<Ctrl-R>/ inserts the last searched for regex between the slashes. This is wonderful and no more copypasting stuff! 1

Intellij idea checkstyle tests

In settings, you can configure checkstyle to run everywhere [not] including tests. If you don’t include tests, you won’t get to ‘run checkstyle on the current file’.

i3 vertical tabbed windows

If I have windows that are tabs stacked vertically to make it not-a-tab-anymore I just have to move it one level above or below tabs. If it’s stack 3/3, amking it 4/3 would make it the bottom part of a split whose top part is vertical tabs.

zsh evil registers

Intellij idea read-only decompiled class sources

.. predictably, “Download sources” makes them downloaded, readable and debuggable.

Also nice is the “Alternative source available for” - todo understand how exactly do poms, sources, etc etc etc. play with all this. I assume if maven module wants X, and X can be found in maven and in the list of dependencies in Intellij, I get this choice?


  1. Simplifying regular expressions using magic and no-magic | Vim Tips Wiki | Fandom ↩︎

Day 617

zsh prompt changes

To continue the idea to make it easier to record my steps, did some changes to the clean oh-my-zsh theme. For this, I made a copy of the theme file /home/%username%/.oh-my-zsh/themes/clean2.zsh-theme, and using zsh: 13 Prompt Expansion added the following:

  • history event number at the end of the prompt, after the time (RPROMPT='%*/%!')
  • Added the last N elements in the path while leaving tilde contraction. (PROMPT='%{$fg[$NCOLOR]%}%B%n%b%{$reset_color%}:%{$fg[blue]%}%B%5c/%b%{$reset_color%} $(git_prompt_info)%(!.#.$) ', the number 5 represents the number of directories to show.)

Now the prompt looks like this:

%username%:~/o/dtb/assets/pics/ (master✗) $                                                             16:14:10/10548

timewarrior :quarter

Timewarrior has many interesting hints, I always missed something with now and previous month - of course :quarter exists.

Day 616

Disabling wifi to use ethernet

This should have been done long ago but discovered this now. If I have both a wiki and ethernet and want to stop connecting to the wifi, disconnecting from the endpoint is much less reliable than disabling wifi alltogether.

fc and history

fc also accepts the history number of the command to edit and run. So fc 1238, for example.

Ideavim map for <Esc>

Quite often, especially in the commit window, I want to use Ideavim’s normal/command mode, but <Esc> gets intercepted by Idea and closes the commit window. My old mapping works though:

imap jj <Esc>

in .ideavimrc.

German

zsh share registers with OS when copypasting

git clone https://github.com/kutsan/zsh-system-clipboard ~/.zsh/plugins/zsh-system-clipboard

Then

source "$HOME/.zsh/plugins/zsh-system-clipboard/zsh-system-clipboard.zsh"

in .zshrc

Now I can yank stuff from the zsh command line and paste them in other applications etc.


  1. ladebalken | Übersetzung Englisch-Deutsch ↩︎

Day 613

Finally fixed the date / time / tztime saga

Looked into the TZ environment variable, it’s the overwritten “Berlin” one. I forgot I had this beauty in .zshrc: export TZ="/usr/share/zoneinfo/Europe/Berlin"

Probably a relict of an old hacky way to set time when I couldn’t do it properly.

Interestingly, even i3status listened to it (how? why? At which point does it access zsh environment variables? Are environment variables zsh-specific and accessible to things started by/within zsh? How does this work? TODO). And interestingly, restarting zsh didn’t get date to output the right date, even with the right timezone set, I had to close and reopen the terminal.

Removed that line, restarted i3, now all times are set right and I can remove timezone info from my timestamps.

Vim insert time and date - updated mappings

imap <leader>d <esc>:put =strftime('%c')<cr>kJA
imap <leader>t <esc>:put =strftime('%H:%M:%S')<cr>kJA

They look like this:

Fr 04 Sep 2020 09:54:59 CEST
09:55:00

Qutebrowser / vim copypaste

If I’m copypasting something from vim with yW it copies the space after the last word. If I then paste it into the search of qutebrowser it won’t find anything if the string in the page doesn’t have a space after it.

git diff CSVs at field level

git diff --color-words="[^[:space:],]+" 1

daff tool for diff-ing CSVs

Daff is a “library for comparing tables, producing a summary of differences”. GitHub - paulfitz/daff: align and compare tables

It actually does a really nice visualization with colours, like that script I wrote a long time ago. Leaves field numbers too. In general it’s awesome.

More options available with the full form: daff diff --id field --padding sparse file1 file2 Has a lot of output formats!

Even ordered/unordered works, if you don’t set the wrong id column.

Vim folds empty line

To make the markers marking folds easier to edit, one can select an empty line at the end of zf, they will end up on a line by themselves, and you can happily paste stuff inside the fold.


  1. Tool to diff CSV files at the field level? - Stack Overflow ↩︎

Day 612

New shiny zsh alias

ff is now find | grep

i3 run and show windows

# bindsym $ms+d exec --no-startup-id i3-dmenu-desktop
bindsym $ms+d exec rofi -show run

# select windown
bindsym $ms+g exec rofi -show window

in i3/config. Rofi is nice and I forgot about it. Added a list of windows on $ms+g so that it’s similar to my qutebrowser keybinding.

Dependencies scopes in Intellij Idea

The default, “Compile”, is the “strongest”. Module dependencies - Help | IntelliJ IDEA contains a really nice table summarizing compile/test/runtime/provided.

Running a single test in maven

mvn -Dtest=UsersServiceImplTest test for all tests in that class, but you can also do wildcards and single test (yourclass#yourtest test) 1

Intellij idea module dependencies order

Order is important and whichever is the first will be used.

You can change the order in the GUI, but you can move them one step above and you can’t move multiple dependencies at the same time. Of course if you add a new one it goes to the veeeery end.

You can edit the configuration manually, inside the project it’s the .iml file that can be edited via whatever.

Get column in bash with cut

This is a thing that keeps coming up, TODO anki

awk - bash: shortest way to get n-th column of output - Stack Overflow

cut -f2 accesses the second field. -d is for delimiters, if not equal to tab.

fex.net

Fex.net seems to be an OK service for sharing stuff, S. used it for wedding pictures and A. used it for uploading an .iso, both worked well.


  1. Running a Single Unit Test with Maven - Apps Developer Blog ↩︎

Day 611

Libreoffice Calc opening files with default import settings

For multiple files with known settings, such as separator, decimal etc., libreoffice --calc mydata.csv --infilter=”CSV:44,34,0,1,4/2/1”

  1. Separator, 44 is the ASCII value of comma
  2. Text delimiter, 34 is the ASCII value of double quote
  3. Character set to use, 0 is the system default
  4. Line number to start conversion. I have no header line, so it is 1
  5. Cell format per column, separated by /. It does not seem to matter if there are a different number of columns in the data: Extra columns default to Standard, extra formats are ignored. 4 = date d/m/y; 2 = text; 1 = standard 1

Full documentation for infilter is here: Filter Options - Apache OpenOffice Wiki

Jekyll mixing html and markdown

I have to remember that putting markdown stuff inside HTML tags won’t work, like with the Actionlist yesterday.

git diff to stdin

Is it possible to git-diff a file against standard input? - Stack Overflow:

echo foo | git diff --no-index -- my_file -

Since git 1.5.1 there’s been an option to diff against stdin - it’s just not been documented

Germany “c/o” with letters and post


  1. libreoffice - How to avoid Text Import form when opening CSV in Calc? - Unix & Linux Stack Exchange ↩︎

Day 610

x-www-browser and qutebrowser

If qutebrowser is not running and I click a link, another instance will be started, and it looks like this: myusername+ 19648 10.8 2.7 6125248 896828 ? Sl 12:10 10:07 /usr/bin/python3 /usr/bin/x-www-browser https://the-link-i-click.ed; I can’t find it easily as the process doesn’t contain the string qutebrowser.

English / German Redewendungen

English / British

dict.cc | analog | English Dictionary “analog” is English spelling, British is “analogue”

NLP / NER / Named entity recognition / ML

CH’s favourite example of ambiguity in NER:

[Paris(LOC) Hilton(ORG)](PER)

zsh theme

Changed it to clean, now I get the time when I ran commands. I so missed this, not sure when it all went wrong.

zsh history with datetime

history -E shows time and date when a command was run.

I really think I need something like a lab notebook approach to all my work experiments, this is a good step for this. So many terminals around with so much stuff run with different parameters and no way to track it.

history -E:

 7932  1.9.2020 18:11  history
 7933  1.9.2020 18:11  vim ~/.zshrc
 7934  1.9.2020 18:11  zsh
 7935  1.9.2020 18:11  ls
 7936  1.9.2020 18:13  pwd
 7937  1.9.2020 18:13  ls
 7938  1.9.2020 18:13  cd newkotlin-newusecase
 7939  1.9.2020 18:13  pwd | xc
 7940  1.9.2020 18:13  ls
 7941  1.9.2020 18:13  history -E
 7942  1.9.2020 18:15  vim ~/.zshrc
 7943  1.9.2020 18:15  zsh

Outputs only the latest items though. But it accepts a start parameter, so history -E 0 gives me what I need.

Also I didn’t fix the time last time apparently, date still disagrees with tztime, whatever I did last time was a temporary fix. Will be printing timezone info starting from now on.

Now I have a couple more default aliases:

alias h='history -E 1 | grep'
alias hh='history -E 1'
alias g='grep'

zsh alias for date in filenames/commands

alias sht='date +"%m%d-%H%M%S%z"'

Now I can do whatevercommand -o outputfile-$(sht) and it will paste a datetime like outputfile-0901-182113+0300 automatically.

curl quiet mode

The magic for waiting for files will be now watch curl -I -s http://somefile | grep 200 interestingly, echo watch "curl ..." | xc which xc being clipboard gave me a string without "s, not sure at which step they got removed but it might bite me in the future.

Real command is watch "curl -I -s http://somefile | grep 200". . -s makes it show no progress bars etc.

Day 609

German

  • nachschlagen - look sth. up, retaliation, consult something quickly. TODO anki
  • massig - bulky, massive, huge. (“Massiges Gehalt”)

Intellij idea split tabs

Intellij Idea has ‘unsplit’ lower than all the other tab options :)

TODO English

Finally remember how many “d"s and “s"s are there in address. TODO anki

Intellij idea reopen closed tab

Like in browsers, it’s <Ctrl-Shift-T>; Also now it’s also u like in qutebrowser: map <leader>u :action ReopenClosedTab<CR>

Edit: Why doesn’t it work? TODO

Intellij idea / pycharm environment variables

You can copypaste the environment variables in the running configuration, including from system ones that get included to the custom ones, it understands what you want.

Ideavim / Intellij idea actionlist

Since I usually look for this here by looking for items from .ideavimrc

Actionlist
$Copy                                              <M-C>
$Cut                                               <M-X> <S-Del>
$Delete                                            <Del> <BS> <M-BS>
$LRU                                              
$Paste                                             <M-V>
$Redo                                              <M-S-Z> <A-S-BS>
$SearchWeb                                         <A-S-G>
$SelectAll                                         <M-A>
$Undo                                              <M-Z>
About                                             
Actions.ActionsPlugin.GenerateToString            
ActivateAntBuildToolWindow                        
ActivateApplicationServersToolWindow              
ActivateChangesToolWindow                          <M-=>
ActivateDatabaseToolWindow                        
ActivateDebugToolWindow                            <M-5>
ActivateDesignerToolWindow                        
ActivateEventLogToolWindow                        
ActivateFavoritesToolWindow                       
ActivateFindToolWindow                             <M-7>
ActivateInspectionToolWindow                      
ActivateMavenProjectsToolWindow                   
ActivateMessagesToolWindow                         <M-0>
ActivateNavBar                                    
ActivatePalette	ToolWindow                        
ActivatePaletteToolWindow                         
ActivateProjectToolWindow                          <M-S-9>
ActivateRunToolWindow                              <M-4>
ActivateStructureToolWindow                       
ActivateTerminalToolWindow                         <M-3>
ActivateTODOToolWindow                             <M-6>
ActivateUIDesignerToolWindow                      
ActiveToolwindowGroup                             
Add.Email.Transport                               
Add.Embedded.Http.Transport                       
Add.Hibernate.Framework                           
Add.Ibatis.Framework                              
Add.JdkTimer                                      
Add.Jdo.Framework                                 
Add.Jms.Transport                                 
Add.OpenSymphonyTimer                             
Add.Toplink.Framework                             
Add.Webflow                                       
AddAllToFavorites                                 
AddAntBuildFile                                   
AddAsSwcLib                                       
AddAttributeAction                                
AddDataSourceFromServer                           
AddFrameworkSupport                               
AddGradleDslPluginAction                          
AddNewComponentAction                             
AddNewFavoritesList                               
AddNewMixinAction                                 
AddNewPageAction                                  
AddNewTabToTheEndMode                             
AddOptionDialogActionGroup                        
AddSourcesContentToSourceMap                      
AddSubtagAction                                   
AddToFavorites                                    
AddToFavoritesPopup                                <A-S-F>
AddToISuite                                       
AddToTestNGSuite                                  
AJ.NewAspect                                      
AJ.PushIn                                         
AlienCommitChangesDialog.AdditionalActions        
AlienPropertiesLocalDiff                          
AnalyzeActions                                    
AnalyzeJavaMenu                                   
AnalyzeMenu                                       
AnalyzeStacktraceOnError                          
Android.ConvertToNinePatch                        
Android.CreateResourceDirectory                   
Android.CreateResourcesActionGroup                
Android.Debugger.ViewBitmapAction                 
Android.EnableDdms                                
Android.GenerateSignedApk                         
Android.GenerateSourcesAction                     
Android.MainToolBarActionGroup                    
Android.MemoryMonitor                             
Android.OpenStringResourceEditor                  
Android.RunAndroidAvdManager                      
Android.RunAndroidSdkManager                      
Android.RunDdms                                   
Android.ShowLicenses                              
Android.ShowNavigationEditor                      
Android.SyncProject                               
Android.TraceViewSearch                            <M-F>
Android.UploadDatabase                            
AndroidConnectDebuggerAction                      
AndroidExtractAsIncludeAction                     
AndroidExtractStyleAction                         
AndroidFindStyleApplicationsAction                
AndroidInlineIncludeAction                        
AndroidInlineStyleReferenceAction                 
AndroidToolsGroup                                 
Annotate                                           <C-G>
AnonymousToInner                                  
AntBuildGroup                                     
AppEngine.UploadApplication                       
ApplyJavaeeStyle                                  
AppServers.ChooseArtifacts                        
AppServers.DebugServer                            
AppServers.DeployAll                              
AppServers.EditServerConfig                       
AppServers.RemoveArtifact                          <Del> <BS> <M-BS>
AppServers.RunServer                              
AppServers.StopServer                             
Arrangement.Alias.Rule.Add                        
Arrangement.Alias.Rule.Context.Menu               
Arrangement.Alias.Rule.Edit                       
Arrangement.Alias.Rule.Match.Condition.Move.Down  
Arrangement.Alias.Rule.Match.Condition.Move.Up    
Arrangement.Alias.Rule.Remove                     
Arrangement.Alias.Rule.ToolBar                    
Arrangement.Custom.Token.Rule.Edit                
Arrangement.Rule.Add                              
Arrangement.Rule.Edit                             
Arrangement.Rule.Group.Condition.Move.Down        
Arrangement.Rule.Group.Condition.Move.Up          
Arrangement.Rule.Group.Control.ToolBar            
Arrangement.Rule.Match.Condition.Move.Down        
Arrangement.Rule.Match.Condition.Move.Up          
Arrangement.Rule.Match.Control.Context.Menu       
Arrangement.Rule.Match.Control.ToolBar            
Arrangement.Rule.Remove                           
Arrangement.Rule.Section.Add                      
AssociateWithFileType                             
AutoIndentLines                                    <A-C-I>
AutoShowProcessWindow                             
AxisActions                                       
Back                                               <M-A-Left> button=4 clickCount=1 modifiers=0 <C-[>
BackgroundTasks                                   
BaseOnThisFunction                                
Bash.NewBashScript                                
Bash.REPL.Group                                   
Batch.Job.Create                                  
Bookmarks                                         
BuildArtifact                                     
BuildMenu                                         
ByteCodeViewer                                    
CallHierarchy                                      <A-C-H>
CallHierarchy.BaseOnThisType                      
CallHierarchyPopupMenu                            
CaptureCPUUsageData                               
CaptureMemorySnapShot                             
ChangeAttributeValueAction                        
ChangeCodeStyleScheme                             
ChangeColorScheme                                 
ChangeFileEncodingAction                          
ChangeInspectionProfile                           
ChangeKeymap                                      
ChangeLaf                                         
ChangeLineSeparators                              
ChangeScheme                                      
ChangeSignature                                    <M-F6>
ChangeSplitOrientation                            
ChangesView.AddUnversioned                         <M-A-A>
ChangesView.AddUnversioned.From.Dialog             <M-A-A>
ChangesView.ApplyPatch                            
ChangesView.Browse                                
ChangesView.Commit                                 <C-G>
ChangesView.CreatePatch                           
ChangesView.CreatePatchFromChanges                
ChangesView.DeleteUnversioned                     
ChangesView.DeleteUnversioned.From.Dialog         
ChangesView.Diff                                   <M-D>
ChangesView.Edit                                  
ChangesView.Ignore                                
ChangesView.Move                                  
ChangesView.NewChangeList                         
ChangesView.Refresh                               
ChangesView.RemoveChangeList                      
ChangesView.RemoveDeleted                         
ChangesView.Rename                                
ChangesView.Revert                                 <M-A-Z>
ChangesView.SetDefault                            
ChangesView.Shelve                                
ChangesViewPopupMenu                              
ChangesViewToolbar                                
ChangeTemplateDataLanguage                        
ChangeTypeSignature                                <M-S-F6>
ChangeView                                        
CheckForUpdate                                    
CheckinFiles                                      
CheckinProject                                    
CheckStatusForFiles                               
ChooseActiveBuildConfiguration                    
ChooseDebugConfiguration                           <A-C-D>
ChooseNextSubsequentPropertyValueEditorAction      <C-Down>
ChoosePrevSubsequentPropertyValueEditorAction      <C-Up>
ChooseRunConfiguration                             <A-C-R>
ClassNameCompletion                                <A-C- >
ClassTemplateNavigation                            <M-S-G>
CleanGradleProject                                
ClearCase.Add                                     
ClearCase.CheckOut                                
ClearCase.DeliverActivities                       
ClearCase.FindProjectCheckouts                    
ClearCase.Hijack                                  
ClearCase.Merge                                   
ClearCase.MergeProject                            
ClearCase.Properties                              
ClearCase.RebaseProject                           
ClearCase.SynchronizeWithActivities               
ClearCase.UndoCheckOut                            
ClearCase.Update                                  
ClearCase.VersionTree                             
ClearCaseFile                                     
ClearCaseProject                                  
CloseActiveTab                                     <C-S-F4>
CloseAllEditors                                   
CloseAllEditorsButActive                          
CloseAllUnmodifiedEditors                         
CloseAllUnpinnedEditors                           
CloseContent                                       <M-W>
CloseEditor                                       
CloseEditorsGroup                                 
CloseProject                                      
CodeCleanup                                       
CodeCompletion                                     <C- >
CodeCompletionGroup                               
CodeEditorBaseGroup                               
CodeEditorViewGroup                               
CodeFormatGroup                                   
CodeInsightEditorActions                          
CodeInspection.OnEditor                            <A-S-I>
CodeMenu                                          
CollapseAll                                        <M-m> <M-->
CollapseAllRegions                                 <M-S-m> <M-S-->
CollapseBlock                                      <M-S-.>
CollapseDocComments                               
CollapseRegion                                     <M-m> <M-->
CollapseRegionRecursively                          <M-A-m> <M-A-->
CollapseSelection                                  <M-.>
CollapseTreeNode                                   <m>
com.ansorgit.plugins.bash.actions.AddReplAction   
com.intellij.execution.testframework.sm.runner.ui.statistics.ShowStatisticsAction
com.intellij.execution.testframework.sm.runner.ui.statistics.ShowTestProxy
com.intellij.play.console.RunPlayConsoleAction    
com.intellij.spellchecker.actions.SpellingPopupActionGroup
com.intellij.sql.refactoring.SqlExtractNamedQueryAction
com.jetbrains.php.framework.FrameworkRunConsoleAction <M-S-X>
com.jetbrains.plugins.remotesdk.console.RunSshConsoleAction <C-Z>
CombinePropertiesFilesAction                      
CommanderPopupMenu                                
CommentByBlockComment                              <M-A-/> <M-A-o> <C-S-/> <C-S-o>
CommentByLineComment                               <M-/> <M-o>
CommentGroup                                      
CommittedChanges.Clear                            
CommittedChanges.Details                          
CommittedChanges.Filter                           
CommittedChanges.Refresh                          
CommittedChanges.Revert                           
CommittedChangesToolbar                           
Compare.LastVersion                                <C-Z>
Compare.SameVersion                               
Compare.Selected                                  
Compare.Specified                                 
CompareActions                                    
CompareClipboardWithSelection                     
CompareDirs                                       
CompareFileWithEditor                             
CompareTwoFiles                                   
Compile                                            <M-S-F9>
CompileCoffeeScript                               
CompileDirty                                       <M-F9>
CompileEcmaScript6                                
CompileProject                                    
CompilerErrorViewPopupMenu                        
ConfigureCvsRoots                                 
Console.Dialect.SpecificGroup                     
Console.EditorTableResult.Group                   
Console.Execute                                    <CR>
Console.Execute.Multiline                          <M-CR>
Console.History.Browse                             <M-A-E>
Console.History.Next                              
Console.History.Previous                          
Console.HistoryActions                            
Console.Jdbc.Close                                 <C-S-F4>
Console.Jdbc.Execute                               <M-CR>
Console.Jdbc.Execute.Selection                    
Console.Jdbc.ExplainPlan                          
Console.Jdbc.ExplainPlan.Raw                      
Console.Jdbc.Terminate                             <M-F2>
Console.JdbcActions                               
Console.Jpa.Close                                  <C-S-F4>
Console.Jpa.Execute                                <M-CR>
Console.Jpa.GenerateDDL                            <M-A-S-CR>
Console.Jpa.GenerateSql                            <M-S-CR>
Console.Jpa.Terminate                              <M-F2>
Console.JpaActions                                
Console.Open                                       <M-S-F10>
Console.Oracle.DbmsOutput                          <M-F8>
Console.SplitLine                                 
Console.TableResult.AddColumn                      <M-S-8>
Console.TableResult.AddRow                         <M-N>
Console.TableResult.Cancel                         <M-F2>
Console.TableResult.CellEditor.Popup              
Console.TableResult.ChangeCellEditorFileEncoding  
Console.TableResult.ChangeCellEditorLanguage      
Console.TableResult.ChangeColumnLanguage          
Console.TableResult.ChooseExtractor               
Console.TableResult.CloneColumn                    <M-A-S-D>
Console.TableResult.CloneRow                       <M-S-D>
Console.TableResult.ColumnActions                 
Console.TableResult.ColumnHeaderPopup             
Console.TableResult.ColumnsList                    <M-F12>
Console.TableResult.ColumnSortAsc                  <A-S-Up>
Console.TableResult.ColumnSortDesc                 <A-S-Down>
Console.TableResult.ColumnSortingActions          
Console.TableResult.ColumnSortReset                <M-A-S-BS>
Console.TableResult.ColumnVisibility               < >
Console.TableResult.CompareWith                   
Console.TableResult.Copy.ChooseExtractorGroup     
Console.TableResult.Copy.ClipboardGroup           
Console.TableResult.Copy.FileGroup                
Console.TableResult.Copy.Settings                 
Console.TableResult.CopyGroup                     
Console.TableResult.CountRows                     
Console.TableResult.Csv.PopupGroup                
Console.TableResult.DeleteColumns                  <A-S-Del>
Console.TableResult.DeleteRows                     <M-BS>
Console.TableResult.EditValue                      <F2> <CR> <A-CR>
Console.TableResult.EditValueMaximized             <S-CR> <A-S-CR>
Console.TableResult.Filter.Custom                 
Console.TableResult.FirstPage                     
Console.TableResult.Group                         
Console.TableResult.Group.Secondary               
Console.TableResult.HideColumn                    
Console.TableResult.LastPage                      
Console.TableResult.LoadFile                      
Console.TableResult.MaximizeEditingCell            <M-A-S-M>
Console.TableResult.NavigateAction                 <M-Down> <F4>
Console.TableResult.NavigateExportedAction         <A-F7> <C-F>
Console.TableResult.NavigateForeignAction          <M-B> button=1 clickCount=1 modifiers=256 button=2 clickCount=1 modifiers=0
Console.TableResult.NavigationAndEditing.Group    
Console.TableResult.NextPage                       <M-A-Down>
Console.TableResult.Options                       
Console.TableResult.Pin                           
Console.TableResult.PopupGroup                    
Console.TableResult.PreviousPage                   <M-A-Up>
Console.TableResult.Reload                         <F5>
Console.TableResult.RenameTab                     
Console.TableResult.ResetView                     
Console.TableResult.SaveLobAs                     
Console.TableResult.SetDefault                     <M-A-D>
Console.TableResult.SetNull                        <M-A-N>
Console.TableResult.ShowQuery                     
Console.TableResult.SortToggle                    
Console.TableResult.SubmitAddedRow                 <M-CR>
Console.TableResult.ToggleFilters                 
Console.TableResult.Transpose                     
Console.TabPopupGroup                             
Console.Transaction                               
Console.Transaction.Autocommit                    
Console.Transaction.Commit                        
Console.Transaction.Rollback                      
ConsoleEditorPopupMenu                            
ConsoleView.ClearAll                              
ConsoleView.FoldLinesLikeThis                     
ConsoleView.PopupMenu                             
ConsoleView.ShowAsJsonAction                      
context.clear                                      <A-S-X>
context.load                                      
context.save                                      
ContextHelp                                       
ConvertContentsToAttributeAction                  
ConvertGroovyToJava                               
ConvertIndentsGroup                               
ConvertIndentsToSpaces                            
ConvertIndentsToTabs                              
ConvertSchemaAction                               
ConvertToInstanceMethod                           
ConvertToMacLineSeparators                        
ConvertToUnixLineSeparators                       
ConvertToWindowsLineSeparators                    
CopyAsPlainText                                   
CopyAsRichText                                    
CopyElement                                        <F5>
CopyPaths                                          <M-S-C>
CopyReference                                      <M-A-S-C>
CopyUrl                                           
Coverage                                           <C-C>
CoverageMenu                                      
Create.CDI.Beans.Xml.File                         
CreateBeanManagedEntityBeanAction                 
CreateCMPField                                    
CreateContainerManagedEntityBeanAction            
CreateDesktopEntry                                
CreateEJBInterceptor                              
CreateEJBRelationship                             
CreateFilterAction                                
CreateJavaeeWebPageAction                         
CreateLauncherScript                              
CreateLibraryFromFile                             
CreateListenerAction                              
CreateMessageBean                                 
CreateResourceBundle                              
CreateRunConfiguration                            
CreateServletAction                               
CreateSingletonBean                               
CreateStatefulBean                                
CreateStatelessBean                               
CreateTaglibFileAction                            
CreateTransferObject                              
CreateWebProject                                  
CutCopyPasteGroup                                 
Cvs.Add                                           
Cvs.BrowseCVSRepository                           
Cvs.Checkout                                      
Cvs.CreateBranch                                  
Cvs.CreateTag                                     
Cvs.DeleteTag                                     
Cvs.Edit                                          
Cvs.Editors                                       
Cvs.GetFromRepository                             
Cvs.Ignore                                        
Cvs.Import                                        
Cvs.MergeAction                                   
Cvs.ToggleOffline                                 
Cvs.UndoAdd                                       
Cvs.Unedit                                        
Cvs.WatchAdd                                      
Cvs.Watchers                                      
Cvs.WatchOff                                      
Cvs.WatchOn                                       
Cvs.WatchRemove                                   
CvsActions                                        
CvsFilePopupGroup                                 
CvsGlobalGroup                                    
Database.OpenDbStorageAction                       <M-Down> <F4>
Database.Settings                                 
DatabaseView.AddActionGroup                       
DatabaseView.AddActionGroupPopup                  
DatabaseView.AddDataSourceFromUrl                 
DatabaseView.AddDataSourcesGroup                  
DatabaseView.AssignColor                          
DatabaseView.ChangeSignature                       <M-F6>
DatabaseView.CloseConnectionAction                 <M-F2>
DatabaseView.CopyAction                            <M-S-D>
DatabaseView.Ddl.AddColumn                        
DatabaseView.Ddl.AddForeignKey                    
DatabaseView.Ddl.AddIndex                         
DatabaseView.Ddl.AddPrimaryKey                    
DatabaseView.Ddl.AddTable                         
DatabaseView.DropAction                            <A-CR> <C-CR>
DatabaseView.Dump.FileGroup                       
DatabaseView.GenerateDDLAction                     <M-S-C>
DatabaseView.ImportDataSources                    
DatabaseView.PropertiesAction                      <M-I>
DatabaseView.ShowDiff                             
DatabaseView.SynchronizeAction                     <M-A-Y>
DatabaseViewPopupMenu                             
DatabaseViewToolbar                               
Debug                                              <A-D>
DebugClass                                         <A-S-D>
Debugger.AddSteppingFilter                        
Debugger.AddToWatch                               
Debugger.AdjustArrayRange                         
Debugger.AutoRenderer                             
Debugger.CustomizeContextView                     
Debugger.CustomizeThreadsView                     
Debugger.EditFrameSource                          
Debugger.EditNodeSource                           
Debugger.EditTypeSource                            <S-F4>
Debugger.EvaluateInConsole                        
Debugger.EvaluationDialogPopup                    
Debugger.FocusOnBreakpoint                        
Debugger.FramePanelPopup                          
Debugger.FreezeThread                             
Debugger.InspectPanelPopup                        
Debugger.InterruptThread                          
Debugger.MarkObject                               
Debugger.PopFrame                                 
Debugger.Representation                           
Debugger.ResumeThread                             
Debugger.ShowLibraryFrames                        
Debugger.ShowReferring                            
Debugger.ThreadsPanelPopup                        
Debugger.Tree.AddToWatches                        
Debugger.Tree.EvaluateInConsole                   
Debugger.ViewAsGroup                              
Debugger.ViewText                                 
Debugger.WatchesPanelPopup                        
DebuggerActions                                   
DebugMainMenu                                     
DecrementWindowHeight                              <M-S-Up>
DecrementWindowWidth                               <M-S-Left>
DelegateMethods                                   
DeleteAttributeAction                             
DeleteTagAction                                   
Diagram.DeselectAll                                <M-A-A>
Diagram.SelectAll                                  <M-A>
Diff.EditorPopupMenu                              
Diff.FocusOppositePane                             <M-Tab>
Diff.FocusOppositePaneAndScroll                    <M-S-Tab>
Diff.HighlightMode                                
Diff.IgnoreWhitespace                             
Diff.KeymapGroup                                   <C-Z>
Diff.NextChange                                    <M-S-]> <C-Right>
Diff.PrevChange                                    <M-S-[> <C-Left>
Diff.SelectedChange                                <M-O>
Diff.ShowDiff                                      <M-D>
Diff.ShowInExternalTool                           
Diff.ShowSettingsPopup                             <M-S-D>
Diff.UpdatedFiles                                  <M-D>
DiffPanel.Toolbar                                 
DirDiffMenu                                       
DirDiffMenu.Deployment.MirrorToLeft               
DirDiffMenu.Deployment.MirrorToRight              
DirDiffMenu.EnableEqual                           
DirDiffMenu.EnableLeft                            
DirDiffMenu.EnableNotEqual                        
DirDiffMenu.EnableRight                           
DirDiffMenu.SetCopyToLeft                         
DirDiffMenu.SetCopyToRight                        
DirDiffMenu.SetDefault                            
DirDiffMenu.SetDelete                             
DirDiffMenu.SetNoOperation                        
DirDiffMenu.WarnOnDeletion                        
DissociateResourceBundleAction                    
dmServer.manage.server.libraries                  
Document2XSD                                      
DomCollectionControl                              
DomCollectionControl.Add                           <Ins>
DomCollectionControl.Edit                          <M-Down> <F4>
DomCollectionControl.Remove                        <Del> <BS> <M-BS>
DomElementsTreeView.AddElement                     <Ins>
DomElementsTreeView.AddElementGroup               
DomElementsTreeView.DeleteElement                  <Del> <BS> <M-BS>
DomElementsTreeView.GotoDomElementDeclarationAction <M-Down> <F4>
DomElementsTreeView.TreePopup                     
Drools.CreateRuleFile                             
DSM.Analyze                                       
DSM.Close                                          <C-S-F4>
DSM.DiggIntoDependenciesAction                    
DSM.GoToNextCycle                                  <F2>
DSM.LimitDependencies                             
DSM.OpenDependencies                              
DSM.OpenInEditor                                   <M-Down> <F4>
DSM.Popup                                         
DSM.ToggleCycles                                  
DSM.ToggleFlattenPackages                         
DSM.ToggleGradient                                
DSM.ToggleModuleGroups                            
DSM.ToggleModules                                 
DSM.TogglenPackages                               
DSM.Toolbar                                       
DumpLookupElementWeights                           <M-A-S-W>
DumpSpringConfiguration                           
DumpThreads                                       
DuplicatesForm.SendToLeft                          <M-1>
DuplicatesForm.SendToRight                         <M-2>
DupLocate                                         
Dvcs.Log.ContextMenu                              
Dvcs.Log.Toolbar                                  
EditAndWatch                                      
EditBookmarksGroup                                
EditBreakpoint                                     <M-S-F8>
EditCommitMessage                                 
EditCreateDeleteGroup                             
EditEntityListenerBindingsAction                  
EditFavorites                                     
EditInterceptorBindingsAction                     
EditMacros                                        
EditMenu                                          
Editor.JSLibrariesMenu                            
Editor.JSLibrariesMenu.LibraryList                
EditorActions                                     
EditorAddOrRemoveCaret                             button=1 clickCount=1 modifiers=576
EditorAddRectangularSelectionOnMouseDrag           button=1 clickCount=1 modifiers=832
EditorBackSpace                                    <BS> <S-BS>
EditorChooseLookupItem                             <CR>
EditorChooseLookupItemCompleteStatement            <M-S-CR>
EditorChooseLookupItemDot                          <C-.>
EditorChooseLookupItemReplace                      <Tab>
EditorCloneCaretAbove                             
EditorCloneCaretBelow                             
EditorCodeBlockEnd                                 <M-A-]>
EditorCodeBlockEndWithSelection                    <M-A-S-]>
EditorCodeBlockStart                               <M-A-[>
EditorCodeBlockStartWithSelection                  <M-A-S-[>
EditorCompleteStatement                            <M-S-CR>
EditorContextBarMenu                              
EditorContextInfo                                  <C-S-Q>
EditorCopy                                         <M-C>
EditorCreateRectangularSelection                   button=2 clickCount=1 modifiers=576
EditorCut                                          <M-X> <S-Del>
EditorCutLineBackward                             
EditorCutLineEnd                                   <C-K>
EditorDecreaseFontSize                            
EditorDelete                                       <Del>
EditorDeleteLine                                   <M-BS>
EditorDeleteToLineEnd                             
EditorDeleteToLineStart                           
EditorDeleteToWordEnd                              <A-Del>
EditorDeleteToWordEndInDifferentHumpsMode         
EditorDeleteToWordStart                            <A-BS>
EditorDeleteToWordStartInDifferentHumpsMode       
EditorDown                                         <Down>
EditorDownWithSelection                            <S-Down>
EditorDuplicate                                    <M-S-D>
EditorDuplicateLines                              
EditorEnter                                        <CR>
EditorEscape                                       <Esc>
EditorGutterPopupMenu                             
EditorGutterVcsPopupMenu                          
EditorHungryBackSpace                             
EditorIncreaseFontSize                            
EditorIndentLineOrSelection                       
EditorIndentSelection                              <Tab>
EditorJoinLines                                    <C-S-J>
EditorKillRegion                                  
EditorKillRingSave                                
EditorKillToWordEnd                               
EditorKillToWordStart                             
EditorLangPopupMenu                               
EditorLeft                                         <Left>
EditorLeftWithSelection                            <S-Left>
EditorLineEnd                                      <M-Right> <C-S-E>
EditorLineEndWithSelection                         <S-End> <M-S-Right>
EditorLineStart                                    <M-Left> <C-S-A>
EditorLineStartWithSelection                       <S-Home> <M-S-Left>
EditorLookupDown                                   <C-Down>
EditorLookupUp                                     <C-Up>
EditorMatchBrace                                  
EditorMoveDownAndScroll                           
EditorMoveDownAndScrollWithSelection              
EditorMoveToPageBottom                             <M-Pagedown>
EditorMoveToPageBottomWithSelection                <M-S-Pagedown>
EditorMoveToPageTop                                <M-Pageup>
EditorMoveToPageTopWithSelection                   <M-S-Pageup>
EditorMoveUpAndScroll                             
EditorMoveUpAndScrollWithSelection                
EditorNextWord                                     <A-Right>
EditorNextWordInDifferentHumpsMode                
EditorNextWordInDifferentHumpsModeWithSelection   
EditorNextWordWithSelection                        <A-S-Right>
EditorPageDown                                     <Pagedown> <End>
EditorPageDownWithSelection                        <S-Pagedown>
EditorPageUp                                       <Pageup> <Home>
EditorPageUpWithSelection                          <S-Pageup>
EditorPaste                                        <M-V>
EditorPasteFromX11                                 button=2 clickCount=1 modifiers=0
EditorPasteSimple                                  <M-A-S-V>
EditorPopupMenu                                   
EditorPopupMenu.GoTo                              
EditorPopupMenu.Run                               
EditorPopupMenu1                                  
EditorPopupMenuDebug                              
EditorPopupMenuDebugJava                          
EditorPreviousWord                                 <A-Left>
EditorPreviousWordInDifferentHumpsMode            
EditorPreviousWordInDifferentHumpsModeWithSelection
EditorPreviousWordWithSelection                    <A-S-Left>
EditorResetFontSize                               
EditorRight                                        <Right>
EditorRightWithSelection                           <S-Right>
EditorScrollBottom                                
EditorScrollDown                                  
EditorScrollDownAndMove                           
EditorScrollLeft                                  
EditorScrollRight                                 
EditorScrollToCenter                              
EditorScrollTop                                   
EditorScrollUp                                    
EditorScrollUpAndMove                             
EditorSelectLine                                   <M-A-W> <M-L>
EditorSelectWord                                   <A-Up> <M-Þ>
EditorSplitLine                                    <M-CR>
EditorStartNewLine                                 <S-CR>
EditorStartNewLineBefore                           <M-A-CR>
EditorSwapSelectionBoundaries                     
EditorTab                                          <Tab>
EditorTabCompileGroup                             
EditorTabPopupMenu                                
EditorTabPopupMenuEx                              
EditorTabsGroup                                   
EditorTextEnd                                      <M-End>
EditorTextEndWithSelection                         <M-S-End>
EditorTextStart                                    <M-Home>
EditorTextStartWithSelection                       <M-S-Home>
EditorToggleActions                               
EditorToggleCase                                   <M-S-U>
EditorToggleColumnMode                             <M-S-8>
EditorToggleInsertState                           
EditorToggleShowIndentLines                       
EditorToggleShowLineNumbers                       
EditorToggleShowWhitespaces                       
EditorToggleStickySelection                       
EditorToggleUseSoftWraps                          
EditorUnindentSelection                            <S-Tab>
EditorUnSelectWord                                 <A-Down> <M-S-Þ>
EditorUp                                           <Up>
EditorUpWithSelection                              <S-Up>
editRunConfigurations                             
EditSelectGroup                                   
EditSelectWordGroup                               
EditSmartGroup                                    
EditSource                                         <M-Down> <F4>
EditSourceInNewWindow                              <S-F4>
EmacsStyleIndent                                  
Emmet                                             
EmmetNextEditPoint                                 <A-C-Right>
EmmetPreview                                      
EmmetPreviousEditPoint                             <A-C-Left>
EmmetUpdateTag                                    
EncapsulateFields                                 
EvaluateExpression                                 <A-F8>
ExcludeFromStubGeneration                         
excludeFromSuite                                  
excludeFromTestNGSuite                            
ExcludeFromValidation                             
Exit                                               <M-Q>
ExpandAll                                          <M-k> <M-]>
ExpandAllRegions                                   <M-]>
ExpandAllToLevel                                  
ExpandAllToLevel1                                  <M-A-j> <M-A-j>
ExpandAllToLevel2                                  <M-A-j> <M-A-j>
ExpandAllToLevel3                                  <M-A-j> <M-A-j>
ExpandAllToLevel4                                  <M-A-j> <M-A-j>
ExpandAllToLevel5                                  <M-A-j> <M-A-j>
ExpandDocComments                                 
ExpandLiveTemplateByTab                            <Tab>
ExpandLiveTemplateCustom                          
ExpandRegion                                       <M-k> <M-S-=>
ExpandRegionRecursively                            <M-A-k> <M-A-=>
ExpandToLevel                                     
ExpandToLevel1                                     <M-j> <M-j>
ExpandToLevel2                                     <M-j> <M-j>
ExpandToLevel3                                     <M-j> <M-j>
ExpandToLevel4                                     <M-j> <M-j>
ExpandToLevel5                                     <M-j> <M-j>
ExpandTreeNode                                     <k>
ExportImportGroup                                 
ExportSettings                                    
ExportTestResults                                 
ExportThreads                                     
ExportToEclipse                                   
ExportToHTML                                      
ExportToTextFile                                   <C-O>
ExpressionTypeInfo                                 <C-S-P>
ExternalJavaDoc                                    <S-F1>
ExternalSystem.AfterCompile                       
ExternalSystem.AfterRebuild                       
ExternalSystem.AfterSync                          
ExternalSystem.AssignRunConfigurationShortcut     
ExternalSystem.AssignShortcut                     
ExternalSystem.AttachProject                      
ExternalSystem.BeforeCompile                      
ExternalSystem.BeforeRebuild                      
ExternalSystem.BeforeRun                          
ExternalSystem.BeforeSync                         
ExternalSystem.CollapseAll                         <M-m> <M-->
ExternalSystem.DetachProject                       <Del> <BS> <M-BS>
ExternalSystem.EditRunConfiguration               
ExternalSystem.ExpandAll                           <M-k> <M-]>
ExternalSystem.GroupTasks                         
ExternalSystem.OpenConfig                          <M-Down> <F4>
ExternalSystem.OpenTasksActivationManager         
ExternalSystem.RefreshAllProjects                 
ExternalSystem.RefreshProject                     
ExternalSystem.RemoveRunConfiguration             
ExternalSystem.RunTask                            
ExternalSystem.ShowInheritedTasks                 
ExternalSystem.ShowSettings                       
ExternalSystem.ToggleAutoImport                   
ExternalSystemView.ActionsToolbar                 
ExternalSystemView.ActionsToolbar.CenterPanel     
ExternalSystemView.ActionsToolbar.LeftPanel       
ExternalSystemView.ActionsToolbar.RightPanel      
ExternalSystemView.BaseProjectMenu                
ExternalSystemView.ProjectMenu                    
ExternalSystemView.RunConfigurationMenu           
ExternalSystemView.TaskActivationGroup            
ExternalSystemView.TaskMenu                       
ExternalToolsGroup                                
ExtractClass                                      
ExtractInclude                                    
ExtractInterface                                  
ExtractJspTagAction                               
ExtractMethod                                      <M-A-M>
ExtractModule                                     
ExtractSuperclass                                 
Faces.CreateFacesConfigAction                     
Faces.CreateJsfAction                             
Faces.Generate.From.Persistence                   
Faces.MainGraphPopup                              
Faces.MainPageGraphPopup                          
Faces.MainPageTreePopup                           
Faces.Navigation.Graph.PageTree.GotoDeclarationAction
Faces.Navigation.Graph.PageTree.GotoSourceAction  
Faces.User.Defined.Faces.Config                   
FavoritesViewPopupMenu                            
FileChooser                                       
FileChooser.Delete                                
FileChooser.GotoDesktop                            <M-D>
FileChooser.GotoHome                               <M-1>
FileChooser.GotoJDK                               
FileChooser.GotoModule                             <M-3>
FileChooser.GotoProject                            <M-2>
FileChooser.NewFile                               
FileChooser.NewFolder                              <M-N>
FileChooser.Refresh                               
FileChooser.ShowHiddens                           
FileChooser.TogglePathShowing                      <M-P>
FileChooserToolbar                                
FileEditor.OpenDataEditor                         
FileMainSettingsGroup                             
FileMenu                                          
FileOpenGroup                                     
FileOtherSettingsGroup                            
FileSettingsGroup                                 
FileStructurePopup                                 <M-F12>
FillParagraph                                     
Find                                               <M-F>
FindInPath                                         <M-S-F>
FindMenuGroup                                     
FindModal                                          <C-F>
FindNext                                           <M-G>
FindPrevious                                       <M-S-G>
FindUsages                                         <A-F7> <C-F>
FindUsagesInFile                                   <M-F7>
FindWordAtCaret                                   
FixDocComment                                     
Flex.Debugger.FilterSwfLoadUnloadMessages         
Flex.NewActionScriptClass                         
Flex.NewFlexComponent                             
FoldingGroup                                      
ForceRunToCursor                                   <M-A-F9>
ForceStepInto                                      <A-S-F7>
ForceStepOver                                      <A-S-F8>
Forward                                            <M-A-Right> button=5 clickCount=1 modifiers=0 <C-S-=> <C-]>
FullyExpandTreeNode                                <j> <M-]>
Gant.NewScript                                    
Generate                                           <M-N>
Generate.Constructor.JavaScript                   
Generate.EventHandler.Actionscript                
Generate.GetAccessor.JavaScript                   
Generate.GetSetAccessor.JavaScript                
Generate.GrailsView                               
Generate.SetAccessor.JavaScript                   
Generate.SetUp.Actionscript                       
Generate.TearDown.Actionscript                    
Generate.TestMethod.Actionscript                  
Generate.ToString.Actionscript                    
GenerateAntBuild                                  
GenerateConstructor                               
GenerateCopyright                                 
GenerateCoverageReport                            
GenerateCreateUI                                  
GenerateDataMethod                                
GenerateDTD                                       
GenerateEJBFinderOrSelectorMethod                 
GenerateEJBReference                              
GenerateEntityListenerMethods                     
GenerateEquals                                    
GenerateExternalization                           
GenerateFilterAction                              
GenerateGetter                                    
GenerateGetterAndSetter                           
GenerateGroup                                     
GenerateInterceptorMethods                        
GenerateJavadoc                                   
GenerateJspFunctionAction                         
GenerateJspTagAction                              
GeneratePattern                                   
GeneratePersistenceContextReference               
GeneratePersistenceMapping                        
GenerateResourceReference                         
GenerateServletAction                             
GenerateSetter                                    
GenerateSetUpMethod                               
GenerateSuperMethodCall                           
GenerateTearDownMethod                            
GenerateTestMethod                                
GenerateXmlTag                                    
Generify                                          
Git.Add                                            <M-A-A> <C-G>
Git.Branches                                       <C-G>
Git.CheckoutRevision                              
Git.CompareWithBranch                             
Git.ContextMenu                                   
Git.CreateNewBranch                               
Git.CreateNewTag                                  
Git.Edit.Global.Message                           
Git.Fetch                                         
Git.Init                                          
Git.Log                                           
Git.Log.ContextMenu                               
Git.Log.DeepCompare                               
Git.Log.Toolbar                                   
Git.LogContextMenu                                
Git.Menu                                          
Git.Merge                                         
Git.Pull                                          
Git.Rebase                                        
Git.Rebase.Abort                                  
Git.Rebase.Continue                               
Git.Rebase.Skip                                   
Git.RepositoryContextMenu                         
Git.Reset                                         
Git.Reset.In.Log                                  
Git.ResolveConflicts                              
Git.Revert                                        
Git.Stash                                         
Git.Tag                                           
Git.Unstash                                       
GitFileActions                                    
Github.Create.Gist                                
Github.Create.Pull.Request                        
Github.Open.Commit.In.Browser                     
Github.Open.In.Browser                            
Github.Rebase                                     
Github.Share                                      
GitRepositoryActions                              
GlobalSettings                                    
Go.NewGoFile                                      
GoFmtFileAction                                    <M-A-S-F>
GoFmtProjectAction                                 <M-A-S-P>
GoGenerateBenchmark                               
GoGenerateTest                                    
GoImportsFileAction                               
GoShowTypeInternalAction                          
GoTestGenerateGroup                               
GotoAction                                         <M-S-A>
GotoBookmark0                                     
GotoBookmark1                                     
GotoBookmark2                                     
GotoBookmark3                                     
GotoBookmark4                                     
GotoBookmark5                                     
GotoBookmark6                                     
GotoBookmark7                                     
GotoBookmark8                                     
GotoBookmark9                                     
GotoChangedFile                                    <M-O>
GoToChangeMarkerGroup                             
GoToClass                                          <M-S-G>
GotoClass                                          <M-O>
GoToCodeGroup                                     
GotoCustomRegion                                   <M-A-.>
GotoDeclaration                                    <M-B> button=1 clickCount=1 modifiers=256 button=2 clickCount=1 modifiers=0
GoToEditPointGroup                                
GoToErrorGroup                                    
GotoFile                                           <M-S-O>
GotoImplementation                                 <M-A-B> button=1 clickCount=1 modifiers=768
GotoLine                                           <A-G>
GoToLinkTarget                                    
GoToMenu                                          
GoToMenuEx                                        
GotoNextBookmark                                  
GotoNextError                                      <F2>
GotoNextIncompletePropertyAction                   <F2>
GotoPreviousBookmark                              
GotoPreviousError                                  <S-F2>
GotoRelated                                        <M-C-Up>
GotoRow                                           
GotoSuperMethod                                    <M-U>
GotoSymbol                                         <M-A-O>
GoToTapestryClass                                 
GoToTapestryTemplate                              
GoToTargetEx                                      
GotoTest                                           <M-C-T>
GotoTypeDeclaration                                <M-S-B> <C-S-B> button=1 clickCount=1 modifiers=320 button=2 clickCount=1 modifiers=64
Gradle.AddGradleDslDependencyAction               
Gradle.ExecuteTask                                
Gradle.GenerateGroup                              
Gradle.ToggleOfflineAction                        
Grails                                            
Grails.Controller                                 
Grails.DomainClass                                
Grails.Filter                                     
Grails.GSP                                        
Grails.Script                                     
Grails.Service                                    
Grails.Shell                                      
Grails.TagLib                                     
Grails.UpdateDependencies                         
Grails.WAR                                        
Graph.ActualZoom                                  
Graph.CommonLayoutGroup                           
Graph.Current.Node.Dependencies.Filter            
Graph.DefaultGraphPopup                           
Graph.Delete                                       <Del> <BS> <M-BS>
Graph.ExportToFile                                
Graph.Faces.DefaultGraphPopup                     
Graph.Faces.OpenSelectedPages                      <M-Down> <F4>
Graph.FitContent                                  
Graph.Layout.BalloonLayouter                      
Graph.Layout.CircularLayouter                     
Graph.Layout.DirectedOrthogonalLayouter           
Graph.Layout.Fit.Content                          
Graph.Layout.HierarchicGroupLayouter              
Graph.Layout.OrganicLayouter                      
Graph.Layout.OrthogonalLayouter                   
Graph.Layout.TreeLayouter                         
Graph.Print                                       
Graph.Print.Preview                               
Graph.Show.Edge.Labels                            
Graph.ShowHideGrid                                
Graph.SnapToGrid                                  
Graph.ZoomIn                                       <k> <=>
Graph.ZoomOut                                      <m> <->
Griffon.UpdateDependencies                        
Groovy.Console                                    
Groovy.Doc.Generating                             
Groovy.Dynamic.CollapseAll                        
Groovy.Dynamic.ExpandAll                          
Groovy.Dynamic.Remove                             
Groovy.Dynamic.Toolbar                            
Groovy.NewClass                                   
Groovy.NewScript                                  
Groovy.Shell                                      
Groovy.Shell.Execute                               <M-CR>
GroovyGenerateGroup1                              
GuiceActionGroup                                  
GuiDesigner.AddComponent                          
GuiDesigner.AddGroup                              
GuiDesigner.AddTab                                
GuiDesigner.ChooseLocale                          
GuiDesigner.ComponentTreePopupMenu                
GuiDesigner.CreateComponent                        <M-N>
GuiDesigner.CreateListener                         <M-S-O>
GuiDesigner.DataBindingWizard                     
GuiDesigner.DecreaseIndent                         <S-Tab>
GuiDesigner.DeleteComponent                        <Del> <BS> <M-BS>
GuiDesigner.DeleteGroup                            <Del> <BS> <M-BS>
GuiDesigner.Duplicate                              <M-S-D>
GuiDesigner.EditComponent                          <F2>
GuiDesigner.EditGroup                              <F2>
GuiDesigner.EditorPopupMenu                       
GuiDesigner.ExpandSelection                        <A-Up> <M-Þ>
GuiDesigner.Flatten                               
GuiDesigner.FormSource                            
GuiDesigner.GenerateMain                          
GuiDesigner.GoToListener                           <M-A-B> button=1 clickCount=1 modifiers=768
GuiDesigner.GroupButtons                          
GuiDesigner.IncreaseIndent                         <Tab>
GuiDesigner.MainToolbarActions                    
GuiDesigner.Morph                                 
GuiDesigner.NewActions                            
GuiDesigner.Pack                                  
GuiDesigner.PaletteComponentPopupMenu             
GuiDesigner.PaletteGroupPopupMenu                 
GuiDesigner.PreviewForm                           
GuiDesigner.PropertyInspectorPopupMenu            
GuiDesigner.QuickJavadoc                           <C-J>
GuiDesigner.ReloadCustomComponents                
GuiDesigner.ResetValue                            
GuiDesigner.ShowComponentTags                     
GuiDesigner.ShowGrid                              
GuiDesigner.ShrinkSelection                        <A-Down> <M-S-Þ>
GuiDesigner.SurroundPopup                          <M-A-T>
GuiDesigner.UngroupButtons                        
GWT                                               
GWT.GenerateCompileReport                         
GWT.GenerateUiHandlerMethod                       
GWT.NewEntryPoint                                 
GWT.NewEventWithHandler                           
GWT.NewModule                                     
GWT.NewRemoteService                              
GWT.NewSerialClass                                
GWT.NewTestCase                                   
GWT.NewUiBinder                                   
Help.JetBrainsTV                                  
Help.KeymapReference                              
HelpMenu                                          
HelpTopics                                        
Heroku.RemoteServers.ShowContainerLog             
Heroku.RemoteServersViewPopup                     
Hg.Init                                           
Hg.Log.ContextMenu                                
Hg.Mq                                             
Hg.MQ.Unapplied                                   
hg4idea.branches                                  
hg4idea.CreateNewBranch                           
hg4idea.CreateNewTag                              
hg4idea.file.menu                                 
hg4idea.Graft.Continue                            
hg4idea.merge.files                               
hg4idea.mq.ShowUnAppliedPatches                   
hg4idea.pull                                      
hg4idea.QDelete                                    <Del> <BS> <M-BS>
hg4idea.QFinish                                   
hg4idea.QFold                                      <A-S-D>
hg4idea.QGoto                                     
hg4idea.QGotoFromPatches                           <A-S-G>
hg4idea.QImport                                   
hg4idea.QPushAction                                <A-S-P>
hg4idea.QRefresh                                   <M-R>
hg4idea.QRename                                   
hg4idea.Rebase.Abort                              
hg4idea.Rebase.Continue                           
hg4idea.resolve.mark                              
hg4idea.run.conflict.resolver                     
hg4idea.tag                                       
hg4idea.updateTo                                  
hg4idea.UpdateToRevision                          
Hibernate.SessionFactorySettings                  
HideActiveWindow                                   <S-Esc>
HideAllWindows                                     <M-S-F12>
HideCoverage                                      
HideSideWindows                                   
HierarchyGroup                                    
HighlightUsagesInFile                              <M-S-F7>
HippieBackwardCompletion                           <A-S-/>
HippieCompletion                                   <A-/>
Hotswap                                           
HtmlAddTableColumnAfter                           
HtmlAddTableColumnBefore                          
HtmlTableCellNavigateDown                          <M-A-S-Down>
HtmlTableCellNavigateLeft                          <M-A-S-Left>
HtmlTableCellNavigateRight                         <M-A-S-Right>
HtmlTableCellNavigateUp                            <M-A-S-Up>
I18nize                                           
IDEACoverageMenu                                  
IdeScriptingConsole                               
IgnoreChoicesGroup                                
Images.EditExternally                              <M-A-F4>
Images.Editor.ActualSize                           <M-o> <M-/>
Images.Editor.ToggleGrid                          
Images.Editor.ZoomIn                               <M-k> <M-]>
Images.Editor.ZoomOut                              <M-m> <M-->
Images.EditorPopupMenu                            
Images.EditorToolbar                              
Images.ShowThumbnails                              <M-C-T>
Images.Thumbnails.EnterAction                      <CR>
Images.Thumbnails.Hide                             <M-W>
Images.Thumbnails.ToggleRecursive                  <A-j>
Images.Thumbnails.UpFolder                         <BS>
Images.ThumbnailsPopupMenu                        
Images.ThumbnailsToolbar                          
Images.ToggleTransparencyChessboard               
ImplementMethods                                   <C-I>
ImportModule                                      
ImportModuleFromImlFile                           
ImportProject                                     
ImportSettings                                    
IncomingChanges.Refresh                           
IncomingChangesToolbar                            
IncrementalSearch                                 
IncrementWindowHeight                              <M-S-Down>
IncrementWindowWidth                               <M-S-Right>
InferNullity                                      
InheritanceToDelegation                           
Inline                                             <M-A-N>
InsertLiveTemplate                                 <M-J>
InspectCode                                       
InspectCodeGroup                                  
IntegrateChangeSetAction                          
IntegrateFiles                                    
IntroduceActionsGroup                             
IntroduceConstant                                  <M-A-C>
IntroduceField                                     <M-A-F>
IntroduceFunctionalParameter                      
IntroduceParameter                                 <M-A-P>
IntroduceParameterObject                          
IntroduceVariable                                  <M-A-V>
InvalidateCaches                                  
InvertBoolean                                     
J2EEGenerateGroup                                 
J2EEViewPopupMenu                                 
JasmineGenerateAfterEachMethodAction              
JasmineGenerateBeforeEachMethodAction             
JasmineGenerateNewSpecAction                      
JasmineGenerateNewSuiteAction                     
JavaCompileGroup                                  
JavaDebuggerActions                               
Javaee.Deploy                                     
Javaee.KeepDeployedAfterDisconnect                
Javaee.RefreshDeploymentStatus                    
Javaee.Undeploy                                   
javaee.UpdateRunningApplication                    <M-F10>
JavaeeEditorGenerateGroup                         
JavaeeMigrationGroup                              
JavaeeRunDeploymentViewToolbar                    
JavaeeRunToolWindowToolbar                        
JavaGenerateGroup1                                
JavaGenerateGroup2                                
JavaMethodHierarchyPopupMenu                      
Javascript.Linters.JSCS.Fix                       
JavaScriptDebugger.HideActionsGroup               
JavaScriptDebugger.HideDomPropertiesAction        
JavaScriptDebugger.OpenUrl                        
JAX-RS.GenerateJavaFromWADL                       
JAX-RS.GenerateWADL                               
JAXB                                              
Jdbc.OpenConsole                                   <M-S-F10>
Jdbc.OpenConsole.CreateNew                        
Jdbc.OpenTableEditor                               <M-Down> <F4>
Jdbc.RunQueryInConsole                             <M-CR>
Jdbc.RunQueryInConsole.Selection                  
Jdbc.RunSqlScript                                  <C-S-R>
jpa.AddJpaProviderSupport                         
Jpa.AssignDataSources                             
Jpa.OpenConsole                                    <M-S-F10>
JpaPatternAction                                  
JpaViewPopupMenu                                  
Jpdl.CreateJpdl                                   
Jpdl.Designer                                     
JS.TypeInfo                                       
JS.TypeScript.Compile                             
JS.TypeScript.Compile.Current                     
JSCallHierarchy.BaseOnThisMethod                  
JSCallHierarchyPopupMenu                          
JSClassHierarchyPopupMenu                         
JSHierarchyPopupMenu                              
JSMethodHierarchy.BaseOnThisMethod                
JSMethodHierarchyPopupMenu                        
JspActions                                        
JsTestFrameworkCodeGeneratorGroup                 
JumpToLastChange                                   <M-S-BS>
JumpToLastWindow                                   <F12>
JumpToNextChange                                  
LangCodeInsightActions                            
LanguageSpecificFoldingGroup                      
LegacyNewAndroidComponent                         
LocalHistory                                      
LocalHistory.PutLabel                             
LocalHistory.ShowHistory                          
LocalHistory.ShowSelectionHistory                 
LogDebugConfigure                                 
LookupActions                                     
Macros                                            
MacrosGroup                                       
MainMenu                                          
MaintenanceAction                                  <M-A-S-/>
MaintenanceGroup                                  
MainToolBar                                       
MainToolBarSettings                               
MakeAllJarsAction                                 
MakeJarAction                                     
MakeModule                                        
MakeStatic                                        
ManageProjectTemplatesAction                      
MarkAsOriginalTypeAction                          
MarkAsPlainTextAction                             
MarkExcludeRoot                                   
MarkFileAs                                        
MarkGeneratedSourceRoot                           
MarkGeneratedSourceRootGroup                      
MarkNotificationsAsRead                           
MarkRootGroup                                     
MarkSourceRootGroup                               
Maven.AddFileAsMavenProject                       
Maven.AddManagedFiles                             
Maven.AfterCompile                                
Maven.AfterRebuild                                
Maven.AlwaysShowArtifactId                        
Maven.AssignShortcut                              
Maven.BaseProjectMenu                             
Maven.BeforeCompile                               
Maven.BeforeRebuild                               
Maven.BeforeRun                                   
Maven.BuildMenu                                   
Maven.CollapseAll                                  <M-m> <M-->
Maven.DependencyGraphMenu                         
Maven.DependencyMenu                              
Maven.DownloadAllDocs                             
Maven.DownloadAllGroup                            
Maven.DownloadAllGroupPopup                       
Maven.DownloadAllSources                          
Maven.DownloadAllSourcesAndDocs                   
Maven.DownloadSelectedDocs                        
Maven.DownloadSelectedSources                     
Maven.DownloadSelectedSourcesAndDocs              
Maven.EditRunConfiguration                         <M-Down> <F4>
Maven.ExecuteGoal                                 
Maven.ExpandAll                                    <M-k> <M-]>
Maven.GenerateGroup                               
Maven.GlobalProjectMenu                           
Maven.GroupProjects                               
Maven.IgnoreProjects                              
Maven.NavigatorActionsToolbar                     
Maven.NavigatorProjectMenu                        
Maven.OpenProfilesXml                             
Maven.OpenSettingsXml                             
Maven.RefactoringGroup                            
Maven.Reimport                                    
Maven.ReimportProject                             
Maven.RemoveManagedFiles                          
Maven.RemoveRunConfiguration                       <Del> <BS> <M-BS>
Maven.RunBuild                                    
Maven.RunConfigurationMenu                        
Maven.ShowBasicPhasesOnly                         
Maven.ShowEffectivePom                            
Maven.ShowIgnored                                 
Maven.ShowSettings                                
Maven.ShowVersions                                
Maven.TasksGroup                                  
Maven.ToggleOffline                               
Maven.ToggleProfile                               
Maven.ToggleSkipTests                             
Maven.Uml.Exclude                                  <S-Del>
Maven.UpdateFolders                               
Maven.UpdateFoldersForProject                     
MaximizeToolWindow                                 <M-S-Þ>
MemberPushDown                                    
MembersPullUp                                     
MethodDown                                         <C-Down>
MethodDuplicates                                  
MethodHierarchy                                    <M-S-H>
MethodHierarchy.BaseOnThisType                    
MethodHierarchy.ImplementMethodAction              <C-I>
MethodHierarchy.OverrideMethodAction               <M-S-O>
MethodHierarchyPopupMenu                          
MethodUp                                           <C-Up>
Migrate                                           
MigrateCvsRoot                                    
MinimizeCurrentWindow                             
ModuleSettings                                    
Move                                               <F6>
MoveAttributeInAction                             
MoveAttributeOutAction                            
MoveEditorToOppositeTabGroup                      
MoveLineDown                                       <A-S-Down>
MoveLineUp                                         <A-S-Up>
MoveModuleToGroup                                 
MoveStatementDown                                  <M-S-Down>
MoveStatementUp                                    <M-S-Up>
MoveTabDown                                        <M-K>
MoveTabRight                                       <M-K>
Mq.Patches.ContextMenu                            
Mq.Patches.Toolbar                                
Mvc.Actions                                       
Mvc.Plugins                                       
Mvc.RunTarget                                      <M-A-G>
Mvc.Upgrade                                       
MyEditorPopupMenu                                 
NavbarPopupMenu                                   
NavBarToolBar                                     
NavBarToolBarOthers                               
NavBarVcsGroup                                    
NewAction                                         
NewAndroidAssetImage                              
NewApplicationComponent                           
NewCfmlFile                                       
NewClass                                          
NewCoffeeScriptFile                               
NewDialog                                         
NewDir                                            
NewElement                                         <M-N>
NewElementInMenuGroup                             
NewElementSamePlace                                <A-C-N>
NewFile                                           
NewForm                                           
NewFormSnapshot                                   
NewFromTemplate                                   
NewGroup                                          
NewGroup1                                         
NewGroupPersistence                               
NewGuiceBindingAnnotation                         
NewGuiceMethodInterceptor                         
NewGuiceModule                                    
NewGuiceProvider                                  
NewGuiceScopeAnnotation                           
NewHtmlFile                                       
NewJavaDiagram                                    
NewJavaScriptFile                                 
NewModule                                         
NewModuleComponent                                
NewModuleInGroup                                  
NewModulesDiagram                                 
NewPackageInfo                                    
NewProject                                        
NewProjectComponent                               
NewProjectFromVCS                                 
NewProjectOrModuleGroup                           
NewPropertyAction                                 
NewScratchFile                                     <M-S-N>
NewStylesheetFile                                 
NewTypeScriptFile                                 
NewXml                                            
NewXmlDescriptor                                  
NextDiff                                           <F7> <C-Down>
NextEditorTab                                      <C-S-Right>
NextOccurence                                      <M-A-Down>
NextParameter                                      <Tab>
NextProjectWindow                                  <M-À>
NextSplitter                                       <A-Tab>
NextTab                                            <M-S-]> <C-Right>
NextTemplateParameter                              <Tab>
NextTemplateVariable                               <Tab> <CR>
ObtainPermanentTicket                             
OnlineDocAction                                   
openAssertEqualsDiff                               <M-D>
OpenEditorInOppositeTabGroup                      
OpenEjbERDiagramAction                            
OpenFile                                          
OpenInBrowser                                     
OpenInBrowserEditorContextBarGroupAction          
OpenInBrowserGroup                                
OpenInSceneBuilder                                
OpenModuleSettings                                 <M-Down> <F4>
OpenPersistenceERDiagramAction                    
OpenProjectGroup                                  
OpenProjectWindows                                
OptimizeImports                                    <A-C-O>
org.jetbrains.plugins.groovy.actions.generate.accessors.GroovyGenerateGetterAction
org.jetbrains.plugins.groovy.actions.generate.accessors.GroovyGenerateGetterSetterAction
org.jetbrains.plugins.groovy.actions.generate.accessors.GroovyGenerateSetterAction
org.jetbrains.plugins.groovy.actions.generate.constructors.GroovyGenerateConstructorAction
org.jetbrains.plugins.groovy.actions.generate.equals.GroovyGenerateEqualsAction
org.jetbrains.plugins.groovy.actions.generate.missing.GroovyGenerateMethodMissingAction
org.jetbrains.plugins.groovy.actions.generate.missing.GroovyGeneratePropertyMissingAction
osgi.bnd.reimport                                 
osgi.bnd.reimport.all                             
osmorc.viewGeneratedManifests                     
OtherMenu                                         
OverrideMethods                                    <M-S-O>
PackageAIR                                        
PackageFile                                        <M-S-F9>
Pageflow.Designer                                 
PairFileActions                                   
ParameterInfo                                      <M-P>
PasteMultiple                                      <M-S-V>
Pause                                             
Perforce.Force.Refresh                            
PerforceDirect.Edit                                <M-A-E>
PerforceDirect.Menu                               
PerforceEnableIntegration                         
PinActiveTab                                      
PinToolwindowTab                                  
PlaybackLastMacro                                 
PlaySavedMacrosAction                             
PluginDeployActions                               
PopupHector                                        <M-A-S-H>
PowerSaveGroup                                    
PreviousDiff                                       <S-F7> <C-Up>
PreviousEditorTab                                  <C-S-Left>
PreviousOccurence                                  <M-A-Up>
PreviousProjectWindow                              <M-S-À>
PreviousTab                                        <M-S-[> <C-Left>
PreviousTemplateVariable                           <S-Tab>
PrevParameter                                      <S-Tab>
PrevSplitter                                       <A-S-Tab>
PrevTemplateParameter                              <S-Tab>
Print                                             
PrintExportGroup                                  
ProductivityGude                                  
ProfilingActions                                  
ProjectViewAnalysisGroup                          
ProjectViewChangeView                              <A-F1>
ProjectViewCompileGroup                           
ProjectViewPopupMenu                              
ProjectViewPopupMenuModifyGroup                   
ProjectViewPopupMenuRefactoringGroup              
ProjectViewPopupMenuRunGroup                      
ProjectViewPopupMenuSettingsGroup                 
PropertiesDiff                                    
PropertyInspectorActions                          
PropertyInspectorActions.AddProperty               <M-N>
PropertyInspectorActions.CommonActions            
PropertyInspectorActions.EditValue                 <CR>
PropertyInspectorActions.RemoveProperty            <Del> <BS> <M-BS>
PsiViewer                                         
PsiViewerForContext                               
PublishGroup                                      
PublishGroup.CompareLocalVsRemote                 
PublishGroup.Download                             
PublishGroup.SyncLocalVsRemote                    
PublishGroup.SyncLocalVsRemoteWith                
PublishGroup.Upload                               
PublishGroup.UploadTo                              <M-A-S-X>
PublishGroupPopupMenu                             
QuickActions                                      
QuickChangeScheme                                  <C-À>
QuickDocCopy                                       <M-C>
QuickEvaluateExpression                            <M-A-F8> button=1 clickCount=1 modifiers=512
QuickImplementations                               <M-Y>
QuickJavaDoc                                       <C-J>
QuickList.Deployment                              
QUnitGenerateNewTestAction                        
QUnitGenerateSetupAction                          
QUnitGenerateTearDownAction                       
RearrangeCode                                     
RecentChangedFiles                                 <M-S-E>
RecentChanges                                      <A-S-C>
RecentFiles                                        <M-E>
refactoring.extract.dependency                     <M-A-M>
refactoring.introduce.property                     <M-A-V>
RefactoringMenu                                   
RefactoringMenu1                                  
RefactoringMenu2                                  
RefactoringMenu4                                  
Refactorings.QuickListPopupAction                  <C-T>
ReformatCode                                       <M-A-L> <A-F>
Refresh                                            <M-R>
Register                                          
RegistrationActions                               
ReleasePermanentTicket                            
ReloadFromDisk                                    
RemoteExternalToolsGroup                          
RemoteHost.NewGroup                               
RemoteHost.NewRemoteItem                           <M-N>
RemoteHostView.CopyPaths                           <M-S-C>
RemoteHostView.CreateFile                         
RemoteHostView.CreateFolder                       
RemoteHostView.EditRemoteFile                     
RemoteHostView.EditSource                          <M-Down> <F4>
RemoteHostView.Rename                              <S-F6>
RemoteHostView.SetPermissions                     
RemoteHostView.ToggleExclusion                    
RemoteHostViewPopupMenu                           
RemoteServers.ChooseServerDeployment              
RemoteServers.ChooseServerDeploymentWithDebug     
RemoteServers.ConnectServer                       
RemoteServers.DisconnectServer                    
RemoteServers.EditDeploymentConfig                
RemoteServers.EditServerConfig                    
RemoteServersViewPopup                            
RemoteServersViewToolbar                          
RemoveFromFavorites                                <C-Del>
RemoveMiddleman                                   
RenameAttributeAction                             
RenameElement                                      <S-F6>
RenameFavoritesList                                <S-F6>
RenameFile                                        
RenameTagAction                                   
ReopenClosedTab                                    <M-S-T>
Replace                                            <M-R>
ReplaceAttributeWithTagAction                     
ReplaceConstructorWithBuilder                     
ReplaceConstructorWithFactory                     
ReplaceInPath                                      <M-S-R>
ReplaceMethodWithMethodObject                     
ReplaceTagWithAttributeAction                     
ReplaceTempWithQuery                              
RepositoryChangesBrowserToolbar                   
Rerun                                              <M-R>
RerunFailedTests                                  
RerunTests                                         <M-C-R> <A-S-R>
ResizeToolWindowDown                               <M-S-Down>
ResizeToolWindowGroup                             
ResizeToolWindowLeft                               <M-S-Left>
ResizeToolWindowRight                              <M-S-Right>
ResizeToolWindowUp                                 <M-S-Up>
Resolve                                           
ResolveAll                                        
REST                                              
RESTClient.RunRequest                             
RestoreDefaultLayout                               <S-F12>
Resume                                             <M-A-R>
RevealIn                                          
RevertUnchanged                                   
RevisionGraph                                     
Run                                                <M-A-S-R> <A-R>
RunClass                                           <C-S-R>
RunConfiguration                                  
RunContextGroup                                   
RunContextPopupGroup                              
RunCoverage                                       
RunGc                                             
RunInspection                                      <M-A-S-I>
RunJsbtTask                                        <A-F11>
RunMenu                                           
Runner.CloseAllUnpinnedViews                      
Runner.CloseAllViews                              
Runner.CloseOtherViews                            
Runner.CloseView                                  
Runner.Focus                                      
Runner.FocusOnStartup                             
Runner.Layout                                     
Runner.MinimizeView                               
Runner.RestoreLayout                              
Runner.View.Close.Group                           
Runner.View.Popup                                 
Runner.View.Toolbar                               
RunnerActions                                     
RunnerLayoutActions                               
RunTargetAction                                    <M-S-F10>
RunToCursor                                        <A-F9>
SafeDelete                                         <M-Del>
SaveAll                                            <M-S>
SaveAsNewFormat                                   
SaveAsTemplate                                    
SaveDocument                                      
SaveFileAsTemplate                                
SaveProjectAsTemplateAction                       
ScopeView.EditScopes                              
ScopeViewPopupMenu                                
ScrollTreeToCenter                                
Seam.Create.Components.Xml                        
Seam.Create.Pageflow                              
Seam.Create.Pages.Xml                             
SearchEverywhere                                  
SelectAllOccurrences                               <M-C-G>
SelectIn                                           <A-F1>
SelectInRemoteHost                                
SelectNextOccurrence                               <M-D>
SendEOF                                            <M-D>
SendFeedback                                      
SendToFavoritesGroup                              
Servers.Deploy                                    
Servers.DeployWithDebug                           
Servers.Undeploy                                  
ServersToolWindowToolbar                          
SetJobsForChangeList                              
SeverityEditorDialog                              
Shelve.KeymapGroup                                
ShelveChanges.UnshelveWithDialog                   <M-S-U>
ShelvedChanges.DeleteDeleted                      
ShelvedChanges.Diff                                <M-D>
ShelvedChanges.ImportPatches                      
ShelvedChanges.Rename                              <S-F6>
ShelvedChanges.Restore                            
ShelvedChanges.ShowHideDeleted                    
ShelvedChanges.Unshelve                           
ShelvedChangesPopupMenu                           
ShelvedChangesToolbar                             
ShelvedChangesToolbarGear                         
Show.Current.Revision                             
ShowAppliedStylesAction                           
ShowBackwardPackageDeps                           
ShowBookmarks                                      <M-F3>
ShowCdiDependenciesPopup                          
ShowColorPicker                                   
ShowContent                                        <C-Down>
ShowDependenciesOnTarget                          
ShowErrorDescription                               <M-F1>
ShowExecutionPoint                                 <A-F10>
ShowFilePath                                       <M-A-F12>
ShowIntentionActions                               <A-CR> <C-CR>
ShowJsbtTasks                                     
ShowLog                                           
ShowModulesDependencies                           
ShowNavBar                                         <M-Up> <A-Home>
ShowPackageCycles                                 
ShowPackageDeps                                   
ShowPackageDepsGroup                              
ShowPopupMenu                                     
ShowProcessWindow                                 
ShowProjectStructureSettings                       <M-;>
ShowPropertiesDiffVsLocal                         
ShowRecentFindUsagesGroup                         
ShowReformatFileDialog                            
ShowRegistry                                      
ShowSeamDependenciesGraph                         
ShowSettings                                       <M-,>
ShowSettingsAndFindUsages                          <M-A-S-F7>
ShowSiblings                                      
ShowTabsInSingleRow                               
ShowTips                                          
ShowUmlDiagram                                     <M-A-S-U>
ShowUmlDiagramPopup                                <M-A-U>
ShowUsages                                         <M-A-F7>
SliceBackward                                     
SliceForward                                      
SmartStepInto                                      <S-F7>
SmartTypeCompletion                                <C-S- >
SMTestRunnerStatistics                            
SMTestRunnerTestsTree                             
SplitHorizontally                                  <M-K>
SplitVertically                                    <M-K>
Spring.Beans.Generate.Action                      
Spring.Beans.Generate.Constructor.Dependency.Action
Spring.Beans.Generate.Setter.Dependency.Action    
Spring.Beans.Generate.Testing.Dependency.Action   
Spring.Create.Context.File                        
Spring.Patterns.ActionGroup                       
SpringGenerateGroup                               
Spy-js.AddLabel                                   
Spy-js.CaptureOnlyEvent                           
Spy-js.CaptureOnlyFile                            
Spy-js.CaptureOnlyStackFile                       
Spy-js.CloseDocument                              
Spy-js.CloseTraceFiles                            
Spy-js.Context.Tree                               
Spy-js.Event.Tree                                 
Spy-js.FileDependencyGraph.CommonLayoutGroup      
Spy-js.FileDependencyGraph.Default                
Spy-js.FileDependencyGraph.LocateEvent            
Spy-js.FileDependencyGraph.SearchFunction         
Spy-js.FileDependencyGraph.Tree                   
Spy-js.FindNextFunctionNextCallFromStack          
Spy-js.FindNextFunctionPreviousCallFromStack      
Spy-js.FindPrevFunctionNextCallFromStack          
Spy-js.FindPreviousFunctionPreviousCallFromStack  
Spy-js.FindThisFunctionNextCallFromStack          
Spy-js.FindThisFunctionPreviousCallFromStack      
Spy-js.GoToFirstFoundOccurence                    
Spy-js.GoToLastFoundOccurence                     
Spy-js.GoToNextFoundOccurence                     
Spy-js.GoToPreviousFoundOccurence                 
Spy-js.Graph.ExportToFile                         
Spy-js.Graph.Layout.Fit.Content                   
Spy-js.Graph.Preview                              
Spy-js.Graph.Print                                
Spy-js.Graph.Show.Edge.Labels                     
Spy-js.JumpToCaller                               
Spy-js.JumpToMappedTrace                          
Spy-js.JumpToPackageFile                          
Spy-js.JumpToSource                               
Spy-js.JumpToTrace                                
Spy-js.MuteEvent                                  
Spy-js.MuteFile                                   
Spy-js.MuteNodeModuleAction                       
Spy-js.MuteNodeModules                            
Spy-js.MuteStackFile                              
Spy-js.RefreshDocument                            
Spy-js.RemoveAndCloseTraceFiles                   
Spy-js.RemoveChildren                             
Spy-js.RemoveNode                                 
Spy-js.SearchFunction                             
Spy-js.SearchFunctionFromStack                    
Spy-js.ShowAppDependencyGraph                     
Spy-js.ShowEventDependencyGraph                   
Spy-js.Stack.Toolbar                              
Spy-js.Stack.Tree                                 
sql.ChangeDialectAction                           
SqlGenerateGroup                                  
StandardMacroActions                              
Start.Use.Vcs                                     
StartStopMacroRecording                           
StartupWizard                                     
StepInto                                           <F7>
StepOut                                            <S-F8>
StepOver                                           <F8>
Stop                                               <M-F2>
StoreDefaultLayout                                
StructuralSearchActions                           
StructuralSearchPlugin.StructuralReplaceAction    
StructuralSearchPlugin.StructuralSearchAction     
StructureViewCompileGroup                         
StructureViewPopupMenu                            
Struts.Generate.Actions.Group                     
Struts2.Create.StrutsXml                          
Subversion.BrowseSVNRepository                    
Subversion.CleanupProject                         
Subversion.Clenaup                                
Subversion.CompareWithBranch                      
Subversion.Copy                                   
Subversion.Create.External                        
Subversion.ImportToSVNRepository                  
Subversion.Lock                                   
Subversion.MarkLocallyDeletedTreeResolved         
Subversion.MarkResolved                           
Subversion.MarkTreeResolved                       
Subversion.MergeFrom                              
Subversion.Relocate                               
Subversion.Resolve                                
Subversion.SetProperty                            
Subversion.Share                                  
Subversion.ShareWholeProject                      
Subversion.ShowProperties                         
Subversion.TogglePropertiesDiff                   
Subversion.Unlock                                 
SubversionFilePopupGroup                          
SubversionGroup                                   
SubversionUpdateActionGroup                       
SurroundWith                                       <M-A-T>
SurroundWithEmmet                                 
SurroundWithLiveTemplate                           <M-A-J>
SwitchApply                                        <A-C-CR>
SwitchBootJdk                                     
SwitchCoverage                                     <M-A-F6>
SwitchDown                                         <A-C-Down>
Switcher                                           <C-Tab> <C-S-Tab>
SwitchLeft                                         <A-C-Left>
SwitchRight                                        <A-C-Right>
SwitchUp                                           <A-C-Up>
SwitchViewActions                                 
Synchronize                                        <M-A-Y>
SynchronizeCurrentFile                            
TabList                                           
TabsAlphabeticalMode                              
TabsPlacementBottom                               
TabsPlacementGroup                                
TabsPlacementLeft                                 
TabsPlacementNone                                 
TabsPlacementRight                                
TabsPlacementTop                                  
TagDocumentationNavigation                         <M-S-D>
TalkToFdb.Flex.Debug                              
TapestryGroup                                     
task.actions                                      
tasks.analyze.stacktrace                          
tasks.and.contexts                                
tasks.close                                        <A-S-W>
tasks.configure.servers                           
tasks.create.changelist                           
tasks.goto                                         <A-S-N>
tasks.group                                       
tasks.open.in.browser                              <A-S-B>
tasks.show.task.description                       
tasks.switch                                       <A-S-T>
tasks.switch.toolbar                               <A-S-T>
tasks.toolbar.group