In the middle of the desert you can say anything you want
Messed up merging/rebasing branches from branches from branches, but needed to merge literally a couple of commits.
So I created a clean branch from master. Then:
As usual, docs exist1 and are quite readable.
… is the best thing since sliced bread, I was skeptical at first but makes editing code in multiple windows so much better!
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.8
sudo apt install python3.8-dev
sudo apt-get install python3.8-venv3venv38 for this; if I source venv38/bin/activate python3 becomes python3.8 by default.python3.8-dev was added after an error I had4 when installing pycocotools, it didn’t find python.h when building.
This describes the process well: Install python3.6+ for local user on remote machine without root access - ~/web/logs
The official documentation: 2. Using Python on Unix platforms — Python 3.9.7 documentation
Basically make altinstall is a safer version that doesn’t overwrite system-wide stuff:
make install can overwrite or masquerade the python3 binary. make altinstall is therefore recommended instead of make install since it only installs exec_prefix/bin/pythonversion.
TL;DR:
./configure --prefix=whatever
make
make altinstall
$PATH:
export PATH=$PATH:/data/sh/tools/python3.8/bin
Just now remembered that when doing CSS stuff it’s sometimes cached, and one needs to <Shift-R> or sth similar. Hugo’s automatic reloading reloads the content/templates/…, but not the CSS!
Explains a lot of what happened the last two days.
Copypasting from the docu5:
{{ .Params.bar }}){{ if or (isset .Params "alt") (isset .Params "caption") }} Caption {{ end }}{{ if or
(isset .Params "alt")
(isset .Params "caption")
}}
Given that Hugo’s markdown considers code as part of a bullet-point if it’s indented two spaces more than the *-bulletpoint’s level, and that I have a tabwidth of 4 and tabs everywhere else and two spaces were a pain…
To apply settings only within a specific directory, add this to ~/.vimrc6:
autocmd BufNewFile,BufRead /home/me/ntb/* set tabstop=4 softtabstop=4 shiftwidth=4 expandtab foldmethod=marker
Notably, for me it didn’t work when the path contained a symlink, had to write it explicitly.
Another option from that SO questiont was placing a ~/.vimrc in that directory7, allowing vim to use it by default, and sourcing the usual global one from the first line. Has security implications, may lead to issues with paths/plugins, didn’t try it.
Looking for indentation stuff for the above lead me here: Tab settings in Vim. Summary: | by Ari Sweedler | Medium
It has this description, copying verbatim:
tabstop: display-only, how many spaces does one \t equal visually?shiftwidth: how many spaces does one level of indentation equal? (shifting commands, formatting, behaviour).softtabstop: how much whitespace to add/remove when pressing tab/backspace?
expandtab: should pressing <Tab> on the keyboard create spaces or a tab character?Highlighting tab-indents is easy, and I had these settings for that:
set listchars=tab:\:\
set listchars+=trail:◦
For spaces it’s harder.
Tried the indentLine plugin8, due to it using the conceal setting I couldn’t see my json-quotes and _ underscores anymore. Setting conceallevel to 1 from 2 helped only for the latter. May get fixed by colorscheme/syntax files with less concealed stuff?
Setting let g:indentLine_concealcursor = '' (by default inc) helps - text is not concealed at all in the cursor line in any of the modes. I see all concealed text and don’t see the guides. I can kinda live with that.
In any case replacing the 's in json is ugly.
Then found this excellent SO answer.
set cursorcolumn cursorline highlight the entire column/row where the cursor is. Which is why I want indentation highlighting 99% of the time!
With my newfound vim knowledge, added this to ~/.vimrc:
autocmd filetype python set cursorcolumn cursorline
But this didn’t satisfy me for the dtb and I kept looking.
Then I found vim-indent-guides9 that changes just the background color. Settings I ended up using:
let g:indent_guides_enable_on_vim_startup = 1
let g:indent_guides_auto_colors = 0
let g:indent_guides_start_level = 2
let g:indent_guides_guide_size = 4
" autocmd VimEnter,Colorscheme * :hi IndentGuidesOdd guibg=darkgrey ctermbg=233
autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=233
ctermbg=233is one of the darkest black-like vim colors, there’s a nice vim colors reference10 online.
At the end, wrapped everything related to DTB and indentst in one nice startup function:
fun! SetDTB()
set tabstop=4 shiftwidth=2 expandtab
foldmethod=marker
set nocursorline nocursorcolumn
let g:indent_guides_auto_colors = 0
let g:indent_guides_start_level = 1
let g:indent_guides_guide_size = 1
autocmd VimEnter,Colorscheme * :hi IndentGuidesEven guibg=blue ctermbg=236
endfu
autocmd BufNewFile,BufRead /home/me/ntb/* :call SetDTB()
python - pyvenv not working because ensurepip is not available - Stack Overflow ↩︎
make error under PythonAPI, python.h No such file or directory · Issue #180 · cocodataset/cocoapi ↩︎
Vim: apply settings on files in directory - Stack Overflow ↩︎
Answer about local .vimrc in Vim: apply settings on files in directory - Stack Overflow ↩︎
Yggdroot/indentLine: A vim plugin to display the indention levels with thin vertical lines ↩︎
nathanaelkane/vim-indent-guides: A Vim plugin for visually displaying indent levels in code ↩︎
Set up Hugo for DTB and partly sth I’ll call NTB, which is non-work stuff.
So far Hugo is 110/10.
Old one for now is here.
TODO:
uglyurls: true in config does exactly this!* for it to be parsed correctly. Another reason to revisit my vim tab settings?'''text seems like a workaround:
This is text
No syntax highlighting
This is text
No syntax highlighting _at all_
~~strikethrough~~ 1I love how intuitive it is - needed a dotfile in tree, tried tree -a, it worked.
setUp() and tearDown() methods in unittests get executed before/after each test method!
The dictionary with the datasets is a global dictionary, which means that you can’t register_coco_instances() in separate unittests in the same file!
This worked:
if Constants.TRAIN_DATASET_NAME in MetadataCatalog.data:
MetadataCatalog.remove(Constants.TRAIN_DATASET_NAME)
MetadataCatalog.remove(Constants.TEST_DATASET_NAME)
DatasetCatalog.remove(Constants.TRAIN_DATASET_NAME)
DatasetCatalog.remove(Constants.TEST_DATASET_NAME)
Through IDE settings one can configure whether one or multiple visual guides are shown, and the actual number of characters is configured through Settings -> Code Style.
Jupyter notebooks + RISE + Reveal.js + a makefile: cornell-cs5785-2021-applied-ml/Makefile at main · kuleshov/cornell-cs5785-2021-applied-ml
Squash commits into one with Git - Internal Pointers (link by SO):
# Merge the last 7 commits into one
git rebase --interactive HEAD~[7]
# Merge the commits from that commit hash
git rebase --interactive 6394dc
In the latest one, the commit hash is “the hash of the commit just before the first one you want to rewrite from.”
Practically, assuming I want to squash together the a ones, I’d do git rebase --interactive B as it’s the one immediately following the ones I need.
commit a1 (latest/newest)
commit a2
commit a3
commit B
commit C
When actually doing the squashing, set squash in front of the commit lines to squash.
In the next screen, leave only the commit message(s) needed.
I love how it uses vim for this! Very interesting way to do an interface.
flake8 file.py shows issues;
black file.py applies black.
black --line-length=79 file.py applies the line length as per PEP8.
Pycharm uses 119 characters as limit, coming from Intellij I think; officially PEP8 recommends 79.
Blau sein = be drunk (heard at work)
My blog takes minutes to be generated, this DTB is not far from it either. I heard Hugo is fast, and I dislike most of what’s on my blog, the logical thing seems to burn it to the ground and start from zero using Hugo.
cd themes
git submodule add https://github.com/chollinger93/ink-free
cd ..
echo theme = \"ink-free\" >> config.toml
hugo new posts/my-first-post.md
puts the file in ./content/posts/my-first-post.md
hugo server -Dhugo -Dconfig.toml supports #comments_index.mdhugo new content/pages/one/two/test-page.md
Nice cheatsheet, not mypy-specific: Type hints cheat sheet (Python 3) — Mypy 0.910 documentation
Otherwise:
x = 1 if True else None, x would be Optional[int]forlen()def f(ints: Iterable[int]) -> List[str]:
return [str(x) for x in ints]
From docu 1:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
When I’m in the directory with the test_xy.py files, running python3 -m unittest runs all of them. I can also do python3 -m unittest test_xy for that file, and python3 -m unittest test_xy.TestXY.test_specific_thing.
Found this, and it’s freaking awesome: Debugging by starting a REPL at a breakpoint is fun
Sample from there:
def make_request():
result = requests.get("https://google.com")
import ipdb; ipdb.set_trace()
There’s the default pdb, there’s ipdb that has to be installed.
Adding
import ipdb; ipdb.set_trace()
anywhere in the code launches a typical debug window that can be used to look into the vars etc.
Just used this for the first time to debug a python program that was running on a remote server and failing, but not locally.
SO much better than print(f"XXX {i}") and friends!
Nice tutorial about its usage: Better Python Debugging With IPDB
n - next line in current method (=“step over”)s - next line of executable code anywhere (=“step into”)c - continue till next breakpointr - continue till function returns (would be nice to learn how to do this in pycharm btw!)a - args - print arguments current function receivedb - adds breakpoint to locations
b filename.py:234b <function>b 123 - line in current fileFull documentation here: 26.2. pdb — The Python Debugger — Python 2.7.18 documentation
My main issue with Enum classes was that serialization is weird, especially if you’re dumping parameters. Tried again, found this: python - Serialising an Enum member to JSON - Stack Overflow
TL;DR class EnumDerivedClass(str, Enum)
import json
from enum import Enum
class LogLevel(str, Enum):
DEBUG = 'DEBUG'
INFO = 'INFO'
print(LogLevel.DEBUG)
print(json.dumps(LogLevel.DEBUG))
print(json.loads('"DEBUG"'))
print(LogLevel('DEBUG'))
will output
LogLevel.DEBUG
"DEBUG"
DEBUG
LogLevel.DEBUG
“Folie überspringen” is a much better way to do what I did with setting a yellow background color - easy to see and worst case scenario it’ll just get skipped
If you run tensorboard on a non-existing folder, you’ll get no feedback about it anywhere?.. No data on Tensorboard itself, nothing useful in CLI.
After some ill-fated undoing of commits, couldn’t find the work of an hour or so.
Guess what: Using Local History to Restore Code Fragments or Deleted Files | The IntelliJ IDEA Blog
I knew about local history for a file, but you can do the same for a directory, through its right-click menu in the Projects view!
I already thought I had set up nvidia-smi and friends (Day 850 | Diensttagebuch (Work journal)), then didn’t use it for months, now when I tried it didn’t work anymore, nvidia-smi said “No devices found”
boltctl showed the device as connected and authorized, prime-select said nvidia was selected, modprobe showed that the correct drivers were used and dkms status had said the correct drivers were installed.
(11:53:23/10181)~/$ dkms status
nvidia, 460.73.01, 5.4.0-73-generic, x86_64: installed
nvidia, 460.73.01, 5.4.0-74-generic, x86_64: installed
(11:53:49/10182)~/$ boltctl
[snip]
● Lenovo ThinkPad Thunderbolt 3 Dock #2
├─ type: peripheral
├─ name: ThinkPad Thunderbolt 3 Dock
├─ vendor: Lenovo
├─ uuid: xxx
├─ status: authorized
│ ├─ domain: domain0
│ └─ authflags: none
├─ authorized: Mo 20 Sep 2021 09:41:16 UTC
├─ connected: Mo 20 Sep 2021 09:41:16 UTC
└─ stored: no
● GIGABYTE GV-N1070IXEB-8GD
├─ type: peripheral
├─ name: GV-N1070IXEB-8GD
├─ vendor: GIGABYTE
├─ uuid: xxx
├─ status: authorized
│ ├─ domain: domain0
│ └─ authflags: none
├─ authorized: Mo 20 Sep 2021 09:42:35 UTC
├─ connected: Mo 20 Sep 2021 09:42:35 UTC
└─ stored: Mo 20 Sep 2021 09:31:09 UTC
├─ policy: manual
└─ key: no
(11:54:54/10188)~/$ lsmod
Module Size Used by
nvidia_uvm 1015808 0
nvidia_drm 57344 1
nvidia_modeset 1228800 1 nvidia_drm
nvidia 34123776 17 nvidia_uvm,nvidia_modeset
(11:55:54/10192)~/$ sudo prime-select query
nvidia
What didn’t work:
What worked:
Offtopic, when I was googling these issues I found my own serhii.net link above on the first page of Google for the key ’“nvidia-smi “no devices were found” authorized’, which is both nice and sad at the same time :)
EDIT: the next morning it didn’t work again. None of the same magic steps in all possible orders. I think it might be an issue with the eGPU or dock or something of that level. The best way to check this would be to do the nuclear option, uninstall all drivers, and install from the beginning, but I think my monthly quota of GPU stuff is full five times over now.
We’re on day 993 (!) of Diensttagebuch! Freaking awesome.
Was creating a requirements.txt for detectron2, official install instructions were:
python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html
Answer specificalyl about this: python - How to format requirements.txt when package source is from specific websites? - Stack Overflow:
requirements.txt format is:
[[--option]...]
<requirement specifier> [; markers] [[--option]...]
<archive url/path>
[-e] <local project path>
[-e] <vcs project url>
<requirements specifier> is:
SomeProject
SomeProject == 1.3
SomeProject >=1.2,<2.0
SomeProject[foo, bar]
SomeProject~=1.4.2
The –option (such as the -f/–find-links) is the same as the pip install options you would use if you were doing pip install from the command line.
Therefore, in requirements.txt it ended up literally as this:
--find-links https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html detectron2
And by the way, detectron2’s own requirements.txt demonstrates nicely part of the above.
My own requirements.txt for CUDA 11.1:
opencv-python==4.2.0.32
# torch 1.9 for cuda 10.2 (for this config https://pytorch.org/get-started/locally/ has no versions in the command
# getting both exact versions from pip freeze
-f https://download.pytorch.org/whl/torch_stable.html
torch==1.9.0+cu111
torchvision==0.10.0+cu111
#torch==1.7.1
#torchvision==0.8.2
# python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.9/index.html
-f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html
detectron2
Best part about ag is that I don’t need to escape anything with its default settings:
pip freeze | ag "(detectron|torch)"
pip freeze | grep "\(detectron\|torch\)"
Suddenly stopped getting readable output.
Fix is to add the env variable JB_DISABLE_BUFFERING, without any value, to the env of the test.
teamcity - no output in console for unittests in pycharm 2017 - Stack Overflow
The documentation about default confg covers all the parameters’ meanings and can be used as reference for that! detectron2.config — detectron2 0.5 documentation
And me dreaming up cfg.MODEL.CHECKPOINT_PERIOD is exactly what they wanted to avoid by disallowing the creation of new keys.
Highlights:
# Number of images per batch across all machines. This is also the number
# of training images per step (i.e. per iteration).
_C.SOLVER.IMS_PER_BATCH = 16
For the second time, discovered that Telegram Cache takes 40gb of disk space.
In the phone’s own menus related to disk space, this was shown as “Pictures” taking 40gb, not the Telegram app and its cache. But this info is exposed through Telegram’s own menus.
Who knew you could combine commands! This is how you start tracking tag1,tag2 starting from the end of the previous span:
$ w track :fill tag1,tag2
Backfilled to 2021-09-15T12:21:41
Tracking "tag1,tag2"
Started 2021-09-15T12:21:41
Current 23:47
Total 0:02:06
TL;DR:
duc index ~/ --fs-exclude fuse.sshfs
duc is about disk space, before running it the index should be built/updated. Usually similar to duc index ~/.
If I have a sshfs mounted somewhere, the process never ends as it tries to index the folder where it’s mounted.
Found some solutions:
duc index ~/ --fs-exclude fuse.sshfs
ext3,ext4./etc/fstab didn’t have the sshfs filesystem, but mount called it fuse.sshfs and this worked!duc index ~/ -e "*somefilename*"
*folder\/file* etc).Bonus: -p shows progress during indexing.
Now I have a new alias in ~/.zshrc:
ducindex() {
duc index "$1" -p --fs-exclude fuse.sshfs
}
cdd CLI alias for CD-ing to directory containing a fileI copypaste a lot the locations of the files from pycharm/intellij to run them from CLI or something similar. Easiest way, because they are focused and I don’t need to focus on the files/project view for that. I can’t find an Action in pycharm/intellij to copypaste only the directory.
Yet another alias for today:
cdd() {
$(dirname "$1")
}
dirname gives the directory, dirname .. | cd and dirname ... | xargs cd don’t work (TODO - why?), so I’m using the zsh thing about “cd to the directory if it’s in a command by itself”.
Now cdd /home/me/wahtever/test.py takes me to /home/me/whatever/ which will saved tens of seconds per year!
Of course tiffsplit1 has a sister tiffcp! Ubuntu Manpage: tiffcp - copy (and possibly convert) a TIFF file
Concatenate N pages into a result.tif:
tiffcp xaaa.tif xaab.tif xabr.tif result.tif
# TODO - this is highlighted yellow
# And this is not
# ... BUT!
# TODO - this is highlighted yellow
# This too, because it's indented one space and logically belongs to the comment above!
I often do <CR>ddkkp or d$kp as a reverse-enter, moving what’s to the right of the cursor on the line above the current one. I’m sure something like this already exists in vim.
Detectron’s Yacs has a github repo with documentation and examples, much better than detectron’s own: rbgirshick/yacs: YACS – Yet Another Configuration System
This works:
comp_cfg.set_new_allowed(True)
comp_cfg['key'] = 'value'
Interesting bit about why it’s not like this by default:
We typically don’t use this so that typo in config file can be noticed. 2
Additionally, this is set per leaf, not per config - you can allow adding stuff to the root but not to its existing children.
And, still, even with comp_cfg.set_new_allowed(True), why can’t I merge_from_list etc. for non-existing keys? (TODO)
log.txt is nice and colorful on CLI, I don’t remember how to let vim interpret the CLI colors but less log.txt works magnificently.
cfg.merge_from_file() doesn’t work with new keys · Issue #2082 · facebookresearch/detectron2
If you are inside a screen, and need to write the entire contents to a file (not just the ones currently visible), this will work:
<C-a> :hardcopy -h <filename>.