Thinking out loud and lab notebook style to help me solve a problem, in this installment - creating representative train/test splits.

Problem

Goal: create a test set that looks like the train set, having about the same distribution of labels.

In my case - classic NER, my training instances are documents whose tokens can be a number of different labels, non-overlapping, and I need to create a test split that’s similar to the train one. Again, splitting happens per-document.

Added complexity - in no case I want tags of a type ending up only in train or only in test. Say, I have 100 docs and 2 ORGANIZATIONs inside them - my 20% test split should have at least one ORGANIZATION.

Which is why random selection doesn’t cut it - I’d end up doing Bogosort more often than not, because I have A LOT of such types.

Simply ignoring them and adding them manually might be a way. Or intuitively - starting with them first as they are the hardest and most likely to fail

Implementation details

My training instance is a document that can have say 1 PEOPLE, 3 ORGANIZATIONS, 0 PLACES.

For each dataset/split/document, I have a dictionary counting how many instances of each entity does it have, then changed it to a ratio “out of the total number of labels”.

{
     "O": 0.75,
     "B-ORGANIZATION": 0.125,
     "I-ORGANIZATION": 0,
     "B-NAME": 0,
     "I-NAME": 0,
 }

I need to create a test dataset with the distribution of these labels as close as the train dataset. In both, say, 3 out of 4 labels should be "O".

So - “which documents do I pick so that when their labels are summed up I get a specific distribution”, or close to it. So “pick the numbers from this list that sum up close to X”, except multidimensional.

Initial algo was “iterate by each training instance and put it in the pile it’ll improve the most”.

Started implementing something to do this in
HuggingFace Datasets , and quickly realized that “add his one training instance to this HF Dataset” is not trivial to do, and iterating through examples and adding them to separate datasets is harder than expected.

“Reading the literature”

Generally we’re in the area of concepts like Subset sum problem / Optimization problem / Combinatorial optimization

Scikit-learn

More usefully, specifically RE datasets, How to Create a Representative Test Set | by Dimitris Poulopoulos | Towards Data Science mentioned sklearn.model_selection.StratifiedKFold.

Which led me to sklearn’s “model selection” functions that have a lot of functions doing what I need! Or almost

API Reference — scikit-learn 1.1.2 documentation

And the User Guide specifically deals with them: 3.1. Cross-validation: evaluating estimator performance — scikit-learn 1.1.2 documentation

Anyway - StratifiedKFold as implemented is “one training instance has one label”, which doesn’t work in my case.

My training instance is a document that has 1 PEOPLE, 3 ORGANIZATIONS, 0 PLACES.

Other places

Dataset Splitting Best Practices in Python - KDnuggets

Brainstorming

Main problem: I have multiple labels/ys to optimize for and can’t directly use anything that splits based on a single Y.

Can I hack something like sklearn.model_selection.StratifiedGroupKFold for this?

Can I read about how they do it and see if I can generalize it? (Open source FTW!) scikit-learn/_split.py at 17df37aee774720212c27dbc34e6f1feef0e2482 · scikit-learn/scikit-learn

Can I look at the functions they use to hack something together?

… why can’t I use the initial apporach of adding and then measuring?

Where can I do this in the pipeline? In the beginning on document level, or maybe I can drop the requirement of doing it per-document and do it at the very end on split tokenized training instances? Which is easier?

Can I do a random sample and then add what’s missing?

Will going back to numbers and “in this train set I need 2 ORGANIZATIONS” help me reason about it differently than the current “20% of labels should be ORGANIZATION”?

Looking at vanilla StratifiedKFold

scikit-learn/_split.py at 17df37aee774720212c27dbc34e6f1feef0e2482 · scikit-learn/scikit-learn

They sort the labels and that way get +/- the number of items needed. Neat but quite hard for me to adapt to my use case.

• numpy.bincount — NumPy v1.23 Manual

OK, NEXT

Can I think of this as something like a sort with multiple keys?..

Can I use the rarity of a type as something like a class weight? Ha, that might work. Assign weights in such a way that each type is 100 and

This feels relevant. Stratified sampling - Wikipedia

Can I chunk them in small pieces and accumulate them based on the pieces, might be faster than by using examples?

THIS looked like something REALLY close to what I need, multiple category names for each example, but ended up being the usual stratified option I think:

python - Split data into train/ test files such that at least one sample is picked for both the files - Stack Overflow

This suggests to multiply the criteria and get a lot of bins - not what I need but I keep moving

Can I stratify by multiple characteristics at once?

I think “stratification of multilabel data” is close to what I need

Found some papers, yes this is the correct term I think

scikit-multilearn

YES! scikit-multilearn: Multi-Label Classification in Python — Multi-Label Classification for Python

scikit-multilearn: Multi-Label Classification in Python — Multi-Label Classification for Python

scikit-multilearn: Multi-Label Classification in Python — Multi-Label Classification for Python:

In multi-label classification one can assign more than one label/class out of the available n_labels to a given object.

This is really interesting, still not EXACTLY what I need but a whole new avenue of stuff to look at

scikit-multilearn: Multi-Label Classification in Python — Multi-Label Classification for Python

The idea behind this stratification method is to assign label combinations to folds based on how much a given combination is desired by a given fold, as more and more assignments are made, some folds are filled and positive evidence is directed into other folds, in the end negative evidence is distributed based on a folds desirability of size.

Yep back to the first method!

They link this lecture explaining the algo: On the Stratification of Multi-Label Data - VideoLectures.NET

That video was basically what I needed

Less the video than the slides, didn’t watch the video and hope I won’t have to - the slides make it clear enough.

Yes, reframing that as “number of instances of this class that are still needed by this fold” was a better option. And here binary matrices nicely expand to weighted stratification if I have multiple examples of a class in a document. And my initial intuition of starting with the least-represented class first was correct

Basic algorithm:

• Get class with smallest number of instances in the dataset
• Get all training examples with that class and distribute them first
• Go to next class

Not sure if I can use the source of the implementation: scikit-multilearn: Multi-Label Classification in Python — Multi-Label Classification for Python

I don’t have a good intuition of what they mean by “order”, for now “try to keep labels that hang out together in the same fold”? Can I hack it to

I still have the issue I tried to avoid with needing to add examples to a fold/Dataset, but that’s not the problem here.

Generally - is this better than my initial approach?

What happens if I don’t modify my initial approach, just the order in which I give it the training examples?

Can I find any other source code for these things? Ones easier to adapt?

Anyway

I’ll implement the algo myself based on the presentation and video according to my understanding.

The main result of this session was finding more related terminology and a good explanation of the algo I’ll be implementing, with my changes.

I’m surprised I haven’t found anything NER-specific about creating representative test sets based on the distribution of multiple labels in the test instances. Might become a blog post or something sometime.

Previously: 220601-1707 Huggingface HF Custom NER with BERT

So you have the various mapping functions, but there’s a set_transform which executes a transform when getitem() is called.

Main classes

If I sent you a link to this you probably want the TL;DR at the bottom

Context

Previously: 220712-2208 Slurm creating modifiable persistent container

Problem: I have a docker image in a private docker registry that needs user/pass.

I need to use it in slurm’s pyxis.

The default srun --container-image .. syntax has no obvious place for a Docker registry user/pass.

Trying to use an image from a private registry does this:

$ srun --mem=16384 -c2 --gres=gpu:v100:2 --container-image comp/myimage:latest

slurmstepd: error: pyxis: child 2505947 failed with error code: 1
slurmstepd: error: pyxis: failed to import docker image
slurmstepd: error: pyxis: printing contents of log file ...
slurmstepd: error: pyxis:     [INFO] Querying registry for permission grant
slurmstepd: error: pyxis:     [INFO] Authenticating with user: <anonymous>
slurmstepd: error: pyxis:     [INFO] Authentication succeeded
slurmstepd: error: pyxis:     [INFO] Fetching image manifest list
slurmstepd: error: pyxis:     [INFO] Fetching image manifest
slurmstepd: error: pyxis:     [ERROR] URL https://registry-1.docker.io/[...] returned error code: 401 Unauthorized

Slurm’s pyxis¹ uses enroot² to do the container magic that includes interfacing with Docker.

enroot is installed on the box, Docker isn’t, I have no root access.

Option/attempt 1: Using enroot config to pass a credentials file

I need to pass through srun configs to enroot, so it can access the docker registry.

To pass credentials to it, create a credentials file in $ENROOT_CONFIG_PATH/.credentials:

# DockerHub
machine auth.docker.io login <login> password <password>

That env var is not set in the base system, set it to /home/me/enroot/ and put the file there - same (no) result.

After googling, found this really detailed thread about the way pyxis handles environment variables: enroot/import.md at master · NVIDIA/enroot Especially this specific comment: pyxis doesn’t use environment variables defined in enroot .env files · Issue #46 · NVIDIA/pyxis

So basically, enroot and pyxis are behaving in opposite ways:

• if a ‘dynamic’ env var is defined in enroot conf files, enroot passes it to the container, but not pyxis
• if it’s not defined in enroot conf files, enroot doesn’t pass it to the container, but pyxis does.

I don’t have write access to the enroot config files, but the $ENROOT_CONFIG_PATH isn’t set there, I should be able to change it. No effect though.

Giving up for now, though that would’ve been the most beautiful solution.

Attempt 2: Get the image separately through enroot

I could use pure enroot to get the docker image, then pass the file to srun.

Run “Docker” Containers with NVIDIA Enroot

To use a oath authentication and a token you would need to sign-up/sign-in and create a token (which you can save for reuse) and then do the container import as,

enroot import 'docker://$oauthtoken@nvcr.io#nvidia/tensorflow:21.04-tf1-py3'

Awesome, let’s create a token and try:

… okay, what’s the address of the docker hub? The hub.docker.com one that’s default and ergo not used anywhere, but I need to pass it explicitly?..

Anyway let’s try to get bitnami/minideb from a public repo to pin the syntax down.

hub.docker.com returned 404s, trial and error led me to docker.io:

[INFO] Querying registry for permission grant
[INFO] Permission granted
[INFO] Fetching image manifest list
[ERROR] Could not process JSON input
curl: (23) Failed writing body (1011 != 4220)

registry-1.docker.io actually asked me for a password!

enroot import 'docker://$token@registry-1.docker.io#bitnami/minideb:latest'
[INFO] Querying registry for permission grant
[INFO] Authenticating with user: $token
Enter host password for user '$token':
[ERROR] URL https://auth.docker.io/token returned error code: 401 Unauthorized

Without providing the token the image gets downloaded! Then I found index.docker.io³ that seems to be the correct one.

Okay, let’s get my private one

me@slurm-box:/slurm/me$ ENROOT_CONFIG_PATH=/home/me/enroot enroot import 'docker://index.docker.io#comp/myimage:latest' 

401 error unauthorized, still ignoring my .credentials or env variable pointing to it.

Docker username only:

enroot import 'docker://mydockerusername@index.docker.io#comp/myimage:latest' 

Asks me for a password and then imports correctly! And creates a file called myimage.sqsh in the current dir.

Woohoo, working way to get docker images from private registry!

$ enroot start myimage.sqsh

enroot-nsenter: failed to create user namespace: Operation not permitted

Okay, so I’m not allowed to start them with enroot - not that I had any reason to.

 srun --mem=16384 -c4 --gres=gpu:v100:2 --container-image ./Docker/myimage.sqsh --container-mounts=/slurm/$(id -u -n)/data:/data --container-workdir /data  --pty bash

Drops me inside a shell in the container - it works!

Next step - using the Docker token.

Docker seems to see it as password replacement, this conflicts with official docus:

# Import Tensorflow 19.01 from NVIDIA GPU Cloud
$ enroot import --output tensorflow.sqsh 'docker://$oauthtoken@nvcr.io#nvidia/tensorflow:19.01-py3'

On further googling - that’s a thing specific for nvcr.io, Docker Hub uses Docker stuff and I use that token as password replacement, period. Okay.

Had issues with mounting stuff as /data by default, but that specific bit is used in the docker image too - used something else.

The Dockerfile also has an ENTRYPOINT and sbin wants something to execute, true can be passed. Couldn’t get this to work, no true means sbin refuses to start, passing true makes it ignore the entrypoint altogether. --[no-]container-entrypoint from docu didn’t help - leaving for later.

Final line:

srun  --mem=16384 -c4 --gres=gpu:v100:2 --container-image ./Docker/myimage.sqsh --container-mounts=/slurm/$(id -u -n)/data:/SLURMdata --container-writable python3 -m trainer_module -i /data/ -o /SLURMdata/Checkpoints/ --config-file /SLURMdata/config.yaml

This:

• makes the image writable, so huggingface and friends can download stuff
• makes /slurm/me/data available as /SLURMdata inside the image;
• passes a config file to it that I have inside /data/config.yaml to the trainer (that accesses it as /SLURMdata/config.yaml)
• runs the training on a dataset inside the directory that the Dockerfile puts inside /data in the image itself (the one that conflicted with mine earlier),
• puts training results in a directory inside /SLURMdata which means it’s available to me after sbin is done in my /slurm/me/data directory.

TODO / for later

• Try again to find a way to use a .credentials file, one command less to run then
• How to run my docker image’s ENTRYPOINT

(More) resources

• Enroot import tutorial/help: enroot/import.md at master · NVIDIA/enroot
• Really detailed about pure-enroot Docker bits, superset of the prev. link: Run “Docker” Containers with NVIDIA Enroot
• pyxis has more detailed usage guides! Especially this one: Usage · NVIDIA/pyxis Wiki
• Extremely detailed/useful discussion about pyxis/enroot environment variables:pyxis doesn’t use environment variables defined in enroot .env files · Issue #46 · NVIDIA/pyxis
• Slurm with Docker and examples of doing Jupyter Notebook: Containers - NHR@KIT User Documentation
• Docker best practices, esp. ENTRYPOINT: Best practices for writing Dockerfiles | Docker Documentation

TL;DR

Two ways I found, passing credentials for the docker registry didn’t work, separately downloading the image and then running it did. Read the entire post if you want details on most of this.

Getting the image:

enroot import 'docker://mydockerusername@index.docker.io#comp/myimage:latest' 

Replace mydockerusername with your docker username, comp with companyname and myimage with the name of the image.

It will ask you for your Docker pass or Personal Access Token.

Will download the image into a *.sqsh file in the current directory or whatever you pass through the -o parameter.

Running the image

srun  --mem=16384 -c4 --gres=gpu:v100:2 --container-image ./Docker/myimage.sqsh --container-mounts=/slurm/$(id -u -n)/data:/SLURMdata --container-writable your_command_to_run

# or - if you are running the thing I'm running - ...

srun  --mem=16384 -c4 --gres=gpu:v100:2 --container-image ./Docker/myimage.sqsh --container-mounts=/slurm/$(id -u -n)/data:/SLURMdata --container-writable python3 -m trainer_module -i /data/ -o /SLURMdata/Checkpoints/ --config-file /SLURMdata/config.yaml

In decreasing order of interest/generality:

• pass the downloaded *.sqsh file to --container-image.
• Environment variables get passed as-is in most cases. If you’d do docker run --env ENV_VAR_NAME, here you’d say ENV_VAR_NAME=whatever srun ... or just export ... it before running and it should work.
• --container-writable is needed to make the filesystem writable, huggingface needs that to write cache files
• --container-mounts
  • are /dir_in_your_fs:/dir_inside_docker_image
  • Make sure the Docker itself doesn’t have anything unexpected located at /dir_inside_docker_image

• DataMeasurementsTool - a Hugging Face Space by huggingface
• Introducing the Data Measurements Tool: an Interactive Tool for Looking at Datasets

Really nice, and the blog post introducing it has a lot of general info about datasets that I found very interesting.

From python - How do I type hint a method with the type of the enclosing class? - Stack Overflow:

If you have a classmethod and want to annotate the return value as that same class you’re now defining, you can actually do the logical thing!

from __future__ import annotations

class Whatever:
	# ...
	@classmethod what(cls) -> Whatever:
		return cls()

It started with writing type hints for a complex dict, which led me to TypedDict, slowly went into “why can’t I just do a dataclass as with the rest”.

Found two libraries:

• pydantic
  • Really nice esp. for validation
  • MyClass(**somedict)
  • Couldn’t find an easy way to parse nested dataclasses from the same dict
• Dataclass Wizard
  • Basically a wrapper to read/save (nested!) dataclasses from dicts / json.
  • Does one thing and does it well!
  • When dumping, converted my field_names to camelcase fieldNames, one can disable that from settings: Extending from Meta — Dataclass Wizard 0.22.1 documentation

Kohen’s Kappa

• sklearn.metrics.cohen_kappa_score — scikit-learn 1.1.1 documentation
• Extremely clear explanation: Cohen’s Kappa Statistic - Statistics How To
• Cohen’s Kappa. Understanding Cohen’s Kappa coefficient | by Kurtis Pykes | Towards Data Science
• Important bits:
  • It includes the likelihood that a correct guess happens randomly
  • Only 2 annotators
  • Compares classification on the same items - so as input it gets y_1 and y_2 of the same length with matching annos

TIL another bit I won’t ever use: 21. for/else — Python Tips 0.1 documentation

This exists:

for a in whatveer:
	a.whatever()
else:
	print("Whatever is empty!")

Found it after having a wrong indentation of an else that put it inside the for loop.

Found at least three:

• portion · PyPI
  • Seems to be more about operations
• GitHub - brentp/interlap: fast, pure-python interval overlap testing
  • More focused on querying
• intervaltree · PyPI
  • Too about querying

Spent hours tracking down a bug that boiled down to:

A if args.sth.lower == "a" else B

Guess what - args.sth.lower is a callable, and will never be equal to a string. So args.sth.lower == "a" is always False.

Of course I needed args.sth.lower().

Python sets have two kinds of methods:

• a.intersection(b) which returns the intersection
• a.intersection_update(b) which updates a by removing elements not found in b.

It calls the function-like ones (that return the result) operators, as opposed to the ‘update_’ ones.

(Built-in Types — Python 3.10.5 documentation)

Given an argument -l, I needed to pass multiple values to it.

python - How can I pass a list as a command-line argument with argparse? - Stack Overflow is an extremely detailed answer with all options, but the TL;DR is:

1. nargs:

parser.add_argument('-l','--list', nargs='+', help='<Required> Set flag', required=True)
# Use like:
# python arg.py -l 1234 2345 3456 4567

2. append:

parser.add_argument('-l','--list', action='append', help='<Required> Set flag', required=True)
# Use like:
# python arg.py -l 1234 -l 2345 -l 3456 -l 4567

Details about values for nargs:

# This is the correct way to handle accepting multiple arguments.
# '+' == 1 or more.
# '*' == 0 or more.
# '?' == 0 or 1.
# An int is an explicit number of arguments to accept.
parser.add_argument('--nargs', nargs='+')

Related, a couple of days ago used nargs to allow an empty value (explicitly passing -o without an argument that becomes a None) while still providing a default value that’s used if -o is omitted completely:

    parser.add_argument(
        "--output-dir",
        "-o",
        help="Target directory for the converted .json files. (%(default)s)",
        type=Path,
        default=DEFAULT_OUTPUT_DIR,
        nargs="?",  
    )

Previously: 220622-1744 Directory structure for python research-y projects, 220105-1142 Order of directories inside a python project

Datasets.

HF has recommendations about how to Structure your repository, where/how to put .csv/.json files in various splits/shards/configurations.

These dataset structures are also ones that can be easily loaded with load_dataset(), despite being CSV/JSON files.

Filenames containing ‘train’ are considered part of the train split, same for ‘test’ and ‘valid’

And indeed I could without issues create a Dataset through ds = datasets.load_dataset(my_directory_with_jsons).

A training that worked on my laptop gets kliled on the slurm node.

• Out-of-Memory (OOM) or Excessive Memory Usage | Ohio Supercomputer Center
• Allocating Memory | Princeton Research Computing

sstat was hard to parse and read, wasn’t sure what I want there.

Find out the CPU time and memory usage of a slurm job - Stack Overflow

• sstat is for running jobs, sacct is for finished jobs
• sacct in its examples told me that column name capitalization doesn’t matter

Ended up with this:

 sacct -j 974 --format=jobid,jobname,maxvmsize,avevmsize,maxrss,averss,maxpages,avepages,avecpu,alloccpus,elapsed,state,exitcode,reqcpufreqmax,reqcpufreqgov,reqmem

For running jobs:

 sstat -j 975 --format=jobid,maxvmsize,avevmsize,maxrss,averss,maxpages,avepages,avecpu,reqcpufreqmax,reqcpufreqgov

(Half can be removed, but my goal was to just get it to fit on screen)

W|A is still the best for conversions: 18081980K in gb - Wolfram|Alpha

Other things I learned:

• You can use suffixes in args like --mem=200G

• --mem=0 should give access to all the memory, doesn’t work for me though

• You can do a task farm to run many instances of the same command with diff params: Slurm task-farming for Python scripts | Research IT | Trinity College Dublin

• Found more helpful places

  • Slurm Resource Manger | Research IT | Trinity College Dublin
  • Automating job submission: SLURM Job Submission with R, Python, Bash | Research Computing Lessons

• Man pages for the commands: Slurm Workload Manager - Man Pages
• Slurm Workload Manager - Quick Start User Guide
• GitHub - NVIDIA/pyxis: Container plugin for Slurm Workload Manager
• Slurm Resource Manger | Research IT | Trinity College Dublin

Things that work for my specific instance:

• ssh-copy-id to log in via public key
• kitty +kitten ssh shamotskyi@v-slurm-login
• sshfs
• set -o vi in ~/.bashrc

Problem: how to install packages to run my stuff

Problem: how to install my python packages?

• There’s no pip and I have no admin rights to install python3-ensurepip
• pyxls that does “containers” is there

Sample from documentation about using pyxls:

srun --mem=16384 -c4 --gres=gpu:v100:2 \
--container-image tensorflow/tensorflow:latest-gpu \
--container-mounts=/slurm/$(id -u -n):/data \
--container-workdir /data \
python program.py

Sadly my code needs some additional packages not installed by default there or anywhere, I need to install spacy language packs etc.

I have a Docker image I can use with everything installed on it, but it’s not on any public registry and I’m not gonna setup one just for this.

Solution - Container that gets saved!

You can start interactive jobs, in this case inside a docker container and it drops you inside a shell:

 srun --mem=16384 -c4 --gres=gpu:v100:2 --container-image tensorflow/tensorflow:latest-gpu --container-mounts=/slurm/$(id -u -n):/data --container-workdir /data --pty bash

Couldn’t add users or install packages because nothing was writeablea, so I open the documentation, find interesting flags there:

--container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH
                              [pyxis] the image to use for the container
                              filesystem. Can be either a docker image given as
                              an enroot URI, or a path to a squashfs file on the
                              remote host filesystem.
--container-name=NAME   [pyxis] name to use for saving and loading the
                        container on the host. Unnamed containers are
                        removed after the slurm task is complete; named
                        containers are not. If a container with this name
                        already exists, the existing container is used and
                        the import is skipped.
--container-save=PATH   [pyxis] Save the container state to a squashfs
                        file on the remote host filesystem.
--container-writable    [pyxis] make the container filesystem writable
      --container-readonly    [pyxis] make the container filesystem read-only

So, I can get an image from Docker hub, save that container locally, and then provide that saved one instead of the image from the registry. Nice.

Or just give it a name, it will reuse it instead of reading it.

I can also make it writable.

=> I can create my own docker image, install everything there, and just go inside it to start trainings?

Final command:

 srun --mem=16384 -c4 --gres=gpu:v100:2 --container-image ./test_saved_path --container-save ./test_saved_path_2 --container-mounts=/slurm/$(id -u -n)/data:/data --container-workdir /data  --container-name my_container_name --container-writable --pty bash

It:

• Opens the container image locally, but more likely - reopens the one under its name
• Opens a shell
• Is writable, any changes I do get saved
• At the end the container itself gets saved in ./test_saved_paths_2, just in case the open-the-named-container-by-name ever fails me.
• As a bonus - I can do stuff to make the container usable, instead of the very raw default settings of the server I have no rights to change.

And a folder that locally I have mounted with sshfs that the docker image also has transparent access to makes the entire workflow fast.

The final solution was:

1. Set up the perfect Container based on the TF docker image
2. Create two scripts, one that just starts the training inside it and one that drops you in a shell in that container. Both based on the command above.

(But I still wonder how the rest are doing it, I can’t believe that’s the common way to run stuff that needs an installed package…)

Magic line:

docker rm -f $(docker ps -aq) && docker volume rm -f $(docker volume ls -q)

Intro

My keyboard setup has always been weird, and usually glued together with multiple programs. Some time ago I decided to re-do it from scratch and this led to some BIG improvements and simplifications I’m really happy about, and want to describe here.

Context: I regularly type in English, German, Russian, Ukrainian, and write a lot of Python code. I use vim, qutebrowser, tiling WMs and my workflows are very keyboard-centric.

TL;DR: This describes my current custom keyboard layout that has:

• only two sub-layouts (latin/cyrillic)
• the Caps Lock LED indicating the active one
• Caps Lock acting both as Ctrl and Escape
• things like arrow keys, backspace accessible without moving the right hand
• Python characters moved closer to the main row

It looks like this¹:

and is available on Github.

How I got into custom keyboard layouts

First, one long summer, I switched to the Dvorak keyboard layout² and loved it.

Then I saw xkcd’s Randall Munroe’s Mirrorboard: A one-handed keyboard layout for the lazy – xkcd. The idea is that it’s easy to repeat with your left hand movements that you do with your right if they are mirrored. This works for blind typing too - if you type l with your right pinky finger, probably your left pinky finger ‘knows’ that reflex as well.

I loved the idea. My right hand usually has either a mouse or a cup of tea in it, and casual left-hand typing without needing to learn an entirely new layout sounded really interesting.

I decided to create such a mirrored layout for Dvorak.

This led me to the topic of customizing xkb keyboard layouts (the Arch wiki describes it very well).

At the end I did create a Dvorak Mirrorboard layout and used it more than expected (for example, image editing is easier if you don’t need to move your hand from the mouse).

But almost immediately I realized the potential of editing layouts and started to add things I needed, like Enter/Backspace, ümläüts and ß etc. - still mirrored, but now not a generic layout anymore. Needing a new name I decided on Pchr8board.

There were N iterations, here’s an old post about one of them: Pchr8board - a mirrored left-hand keyboard layout for Dvorak - serhii.net.

Then I kept adding stuff, in the process abandoning most left-hand features. Slowly we converged to a layout I liked.

Non-xkb weirdness

I had other non-standard changes I was really attached to:

• Since forever I have my Caps Lock key remapped to Ctrl, which I strongly recommend to literally everyone. Ctrl is used often and that position is much easier to reach, and no one needs Caps Lock. An ugly xmodmap hack on autostart remapped both keys.
• Caps-Lock-but-now-Ctrl, if released quickly, it acted as Escape. Incredibly neat, for vim especially. I used xcape³ for this.

It all worked but not flawlessly

All together the setup was a net positive, but was very brittle.

Xcape is clearly abandoned, and neither xmodmap nor xcape work in Wayland. But there are far worse problems.

You need to run it manually on startup (could never get it to run automatically in a reliable way, believe me I tried) and every time you connect a new keyboard.⁴

Sometimes it took multiple attempts to get all parts working. And every time you run xmodmap it resets your layout and you need to re-run setxkbmap.

Not a hypothetical scenario

…Which you may not be able to do, because you’re stuck with a broken layout or Caps Lock on or a Ukrainian layout and no way to change it, because you can’t open a terminal and type a command to do that.

Also all the WM keybindings relying on the former Ctrl key are broken in the process.

You can GUESS where the Alt key is now and then try to get into a tty. Then your Ctrl and Esc are not where you are used to.

Long story short - it was worth the pain, but the pain was there. The setup was band aids on top of other band aids, some applied at the very beginning when I had no idea what I was doing but hey, it works.

One keyboard layout to rule them all

Then I finally did it all from scratch and for real.

The layout in all its glory

Only the changed keys are labelled, with the exception of the default characters when they help me to identify a key.

The keys are read like this:

The key change is that the Left Alt button becomes a modifier key that makes more symbols/actions available. “Latch” is <Level3> and is located on Left Alt⁵.

For example, to get Ä you press <Shift+Latch+a>. For ä you just press <Latch+a>. A needs only Shift.

In the layout definition itself this is represented like this:

key <AC01> { [	    a,	A, adiaeresis,	Adiaeresis]	};

The Right Alt key still works like a normal Alt.

Notation

Notation that I made up⁶:

• Written in full (‘Shift’ or ‘Left Alt’) or capitalized (<LALT>) are physical keys on the keyboard. Given as:
  • keylabel / what xkb calls them inside the layout file (<RCTRL>)
  • default Dvorak value (<c> refers to the key that produces an i in QWERTY or ш in Ukrainian/Russian).
• Shift/LALT are the logical thing after all remappings and modifiers are applied
• <Shift+q> are keybindings, with modifiers given by their logical/remapped values (Shift can be located anywhere on the keyboard as long as it works as a Shift), and the letters are the usual/normal/Level1 unchanged ones (<Shift+q> produces a Q, but alone the key <q> would be a lowercase q).

So <LCTRL> would be “left physical key on the keyboard with Ctrl written on it”, <Shift-Latch-g> would be “Press whatever keys / pedals / mouse buttons that are your Shift and Latch modifiers, then the key on the keyboard that in dvorak on normal systems results in a q appearing on the screen”.

Two languages instead of four and one LED

I created two layouts, v6 the latin one with umlauts and everything else, and ruua, that contains both Russian and Ukrainian characters in the same layout.

Pressing the right Control key once changes the layout:

	key  <RCTL> {	[ISO_Next_Group]	};

Having only two layouts means you never have to guess which one comes next or set up indicators in the taskbar. You just let your muscle memory automatically do its thing.⁷

But unlocking the laptop was still a pain. You don’t know the language you were typing in when you locked it, and things like i3lock don’t tell you the layout by default - and you never know if a wrong password is a typo or a wrong layout.

The grp_led option allows you to use keyboard LEDs as indicators.

setxkbmap -option -option 'grp_led:caps' vv,ruua

Now anytime I’m typing in Russian or Ukrainian the Caps Lock LED is on, regardless of what is shown on the display. Pressing RCTRL changes the layout and makes the LED turn off, and you know it worked.

Custom modifiers defined in the layout itself

No xmodmap anymore! Caps Lock is now Ctrl, with Latch it becomes an Escape (and the former Ctrl button is a new modifier key Hyper_L, guaranteed not to collide with anything).

That took time to get right, the key was making Caps Lock a four-level key⁵, then we can define what happens to it with Level3/Latch/<LALT>:

key <CAPS> { type[Group1] = "FOUR_LEVEL", symbols[Group1] = [ Control_L, Control_L, Escape, NoSymbol] };
modifier_map Control { <CAPS> };

For more, look into modifier_map⁸ and real/virtual modifiers⁹ on the Arch Linux Wiki.

Arrow keys and Backspace easy to reach

Shown in purple, directly in the right hand resting position:

• Arrow keys (CHTN is the new WASD!).¹⁰
• <Backspace> and <Delete>!

key <AD10> { [	    l,	L, BackSpace, Delete		]	};
key <AC07> { [	    h,	H,	Left,	Left		]	};
key <AC08> { [	    t,	T,	Down,	Down   ]	};
key <AC09> { [	    n,	N,	Right,	Right		]	};

Being able to quickly delete text with my ring finger without stopping to type to reach the backspace key feels as good as it sounds.

Best thing, all this works with keyboard shortcuts! <Ctrl-Alt-R> deletes the entire previous word, etc.

Programming features

Mostly the improvements cluster in two areas:

• Move all brackets closer to resting position.
  • ALL OF THEM
• No Shift for frequent characters
  • Python and programming:+,-,=
  • Vim and vim-like things: :!
    • ; now needs a Shift, a sacrifice I’m ready to make.

Left Alt as modifier key works very well for them - it’s easier to reach for my left thumb than Shift was for any finger ever.

Some redundancy and left-hand features

There are two additional Enter keys, one on Space and the other one under Escape. Both closer than the real one, and the latter needs only the left hand. (I found I need a left-hand Enter more often than any other.)

On that same tilde key there’s a Compose key too, which allows to type some exotic characters that are used too rarely to get their own key.

There is also more than one way to do slashes, this mostly has to do with old layouts I had and still remember if I’m tired or stressed.

Installing

After you read ArchWiki’s Precautions and preparations and assuming you need both the latin and cyrillic layouts:

• Copy the source files to /usr/share/X11/xkb/symbols/. (Or maybe create a symlink to a version-controlled version of the layout, then you can do your modifications and test them more easily.)
• Name them something reasonable, the file name will be the name used by setxkbmap to refer to the layout.¹¹

For the full experience

1. Assuming the layouts are in /usr/share/X11/xkb/symbols/v6 and /usr/share/X11/xkb/symbols/ruua, run:

   setxkbmap -option -option 'grp_led:caps' v6,ruua
   

2. If it works, add that command to autostart, it should work.

Light-mode experience

If you don’t want to go all-out:

1. Run setxkbmap -option us, now you have it in your terminal history
2. setxkbmap -option -option 'grp_led:caps' v6,us would give you the new layout and on <RCTRL> you get a standard QUERTY one.
3. If something goes wrong, use the arrow keys to find the command setxkbmap -option us and press Enter to run it, and you’re back in known territory.

The layouts definitions

The sources, keyboard-layout-editor.com .json and the pictures are all available on Github. Pasting them below too for completeness and redundancy.

// My current layout, no connection to dvorak-mirrorboard anymore

default  partial alphanumeric_keys modifier_keys
xkb_symbols   "sh" {

	name[Group1] = "SH Custom layout";

	// Using L-Alt as modifier instead of Caps lock.
	key <LALT> { type[Group1] = "ONE_LEVEL", symbols[Group1] = [ ISO_Level3_Shift ] };

	//// TAB AND FRIENDS  
	// Mod+Space is return
	// TODO
	key <SPCE> { [ space, space, Return ] };

	// Bsp, Enter, **Compose Key **
	key <TLDE> {	[     BackSpace,	Multi_key,	Return,	 NoSymbol]	};

	// Tab, LTab, /, b\

	key  <TAB> {	[ Tab,	backslash, slash, NoSymbol]	};

	// Switch groups by RCTL
	key  <RCTL> {	[ISO_Next_Group]	};

	// Caps is Ctrl, ? <Escape> ?
	// Mapping Escape to Caps+Shift doesn't work for some reason
	key <CAPS> { type[Group1] = "FOUR_LEVEL", symbols[Group1] = [ Control_L, Control_L, Escape, NoSymbol] };
    modifier_map Control { <CAPS> };

	key <LCTL> { type[Group1] = "ONE_LEVEL", symbols[Group1] = [Hyper_L] };
	modifier_map Mod3 { Hyper_L };

	////

	//// FIRST ROW 
	// '"`?
	key <AD01> { [  apostrophe,	quotedbl, quoteleft, NoSymbol] };
	// ,<[?
	key <AD02> { [	comma,	less,   bracketleft, NoSymbol] };
	// .>]?
	key <AD03> { [      period,	greater, bracketright, NoSymbol] };
	////

	key <AD04> { [	    p,	P, asciitilde, NoSymbol		]	};
	key <AD05> { 
		[y,	Y, f, F], 
		[a, a, a, a] 
	};

	// Umlauts
	key <AC01> { [	    a,	A, adiaeresis,	Adiaeresis]	};
	key <AC02> { [	    o,	O, odiaeresis,	Odiaeresis]	};
	key <AC03> { [	    e,	E, ediaeresis,	Ediaeresis]	};
	key <AC04> { [	    u,	U, udiaeresis,	Udiaeresis]	};
	key <AC05> { [	    i,	I, d, D		]	};

	key <AB01> { [   colon,	semicolon,z, Z] };
	key <AB02> { [	    q,	Q, v, V		]	};
	key <AB03> { [	    j,	J, w, W		]	};
	key <AB04> { [	    k,	K, m, M		]	};
	key <AB05> { [	    x,	X, b, B		]	};

	key <AE01> {	[	  1,	exclam,		NoSymbol,	NoSymbol	]	};

	// 2@<{
	key <AE02> {	[	  2,	at,		less,	NoSymbol	]	};
	// 3#>}
	key <AE03> {	[	  3,	numbersign,	greater,	NoSymbol	]	};
	key <AE04> {	[	  4,	dollar,		EuroSign,	NoSymbol	]	};
	key <AE05> {	[	  5,	percent,	NoSymbol,	NoSymbol	]	};

	//// Backspace, arrow keys, ...
	// TODO 
	// key <AD07> { [	    g,	G, Prior, NoSymbol		]	};
	key <AD07> { [	    g,	G, parenleft, braceleft		]	};
	key <AD08> { [	    c,	C,	Up,	 Up	]	};
	key <AD09> { [	    r,	R,	parenright,	braceright		]	};
	// key <AD09> { [	    r,	R,	Next,	Next		]	};
	key <AD10> { [	    l,	L, BackSpace, Delete		]	};
	key <AC07> { [	    h,	H,	Left,	Left		]	};
	key <AC08> { [	    t,	T,	Down,	Down   ]	};
	key <AC09> { [	    n,	N,	Right,	Right		]	};

	key <AD06> { [	    f,	F  		]	};
	// Slash and Backslash
	key <AD11> { [	slash,	question, backslash, NoSymbol	]	};
	key <AD12> { [	equal,	plus		]	};


	// TODO
	key <AC06> { [	    d,	D, NoSymbol, NoSymbol		]	};
    key <AC10> { [	    s,	S,	ssharp,	ssharp		]	};
	key <AC11> { [	minus,	underscore	]	};

	key <AB06> { [	    b,	B		]	};
	key <AB07> { [	    m,	M		]	};
	key <AB08> { [	    w,	W		]	};
	key <AB09> { [	    v,	V		]	};
	key <AB10> { [	    z,	Z		]	};

	// +|\? - the key that by default has only backslash+bar
	
	key <BKSL> { [  plus,  bar, backslash, NoSymbol             ]       };


	key <AE06> {	[	  6,	asciicircum	]	};
	key <AE07> {	[	  7,	ampersand	]	};
	key <AE08> {	[	  8,	asterisk	]	};
	key <AE09> {	[	  9,	parenleft	]	};
	key <AE10> {	[	  0,	parenright	]	};
	key <AE11> {	[     bracketleft,	braceleft	]	};
	key <AE12> {	[     bracketright,	braceright		]	};

};

// Keyboard layouts for Russia.
// AEN <aen@logic.ru>
// 2001/12/23 by Leon Kanter <leon@blackcatlinux.com>
// 2005/12/09 Valery Inozemtsev <shrek@altlinux.ru>
// 2018/07/15 @a13 (a.k.a. @dbvvmpg) and Stepanenko Andrey <ftvkyo2011@yandex.ru>
// 2021 - Adapted to contain Ukrainian characters - serhii.net

// Windows layout
default  partial alphanumeric_keys
xkb_symbols "winkeys" {

    include "ruua(ruua)"
    name[Group1]= "Russian";

    key <AE03> { [           3,  numerosign  ] };
    key <AE04> { [           4,   semicolon  ] };
    key <AE05> { [           5,     percent  ] };
    key <AE06> { [           6,       colon  ] };
    key <AE07> { [           7,    question  ] };
    key <AE08> { [           8,    asterisk, U20BD  ] };

    key <AB10> { [      period,       comma  ] };

    // SH -- now adding the bksp and stuff and removing the Enter thing.
 
	key <SPCE> { [ space] };
	// Mod+Tab gives a slash, which I use often (searching etc.) 
	// Mod+Shift+Tab gives an umlaut on the next character
};

hidden partial alphanumeric_keys
xkb_symbols "ruua" {

    key <AE01> { [           1,      exclam  ] };
    key <AE02> { [           2,    quotedbl  ] };
    key <AE03> { [           3,  numbersign  ] };
    key <AE04> { [           4,    asterisk  ] };
    key <AE05> { [           5,       colon  ] };
    key <AE06> { [           6,       comma  ] };
    key <AE07> { [           7,      period  ] };
    key <AE08> { [           8,   semicolon  ] };
    key <AE09> { [           9,   parenleft  ] };
    key <AE10> { [           0,  parenright  ] };
    key <AE11> { [       minus,  underscore  ] };
    key <AE12> { [       equal,        plus  ] };
    key <BKSL> { [   slash,         backslash  ] };

    key <AB10> { [       slash,    question  ] };
    key <LSGT> { [       slash,         bar  ] };

    key <TLDE> { [       Cyrillic_io,	apostrophe,	U02BC,       Cyrillic_IO  ] };
    key <AD01> { [   Cyrillic_shorti,   Cyrillic_SHORTI  ] };
    key <AD02> { [      Cyrillic_tse,      Cyrillic_TSE  ] };
    key <AD03> { [        Cyrillic_u,        Cyrillic_U  ] };
    key <AD04> { [       Cyrillic_ka,       Cyrillic_KA  ] };
    key <AD05> { [       Cyrillic_ie,       Cyrillic_IE] };
    key <AD06> { [       Cyrillic_en,       Cyrillic_EN  ] };
    key <AD07> { [      Cyrillic_ghe,      Cyrillic_GHE  ] };
    key <AD08> { [      Cyrillic_sha,      Cyrillic_SHA  ] };
    key <AD09> { [    Cyrillic_shcha,    Cyrillic_SHCHA  ] };
    key <AD10> { [       Cyrillic_ze,       Cyrillic_ZE  ] };
    key <AD11> { [       Cyrillic_ha,       Cyrillic_HA  ] };
    key <AD12> { [ Cyrillic_hardsign,	Cyrillic_HARDSIGN,	Ukrainian_yi,	Ukrainian_YI] };

    key <AC01> { [       Cyrillic_ef,       Cyrillic_EF  ] };
    key <AC02> { [     Cyrillic_yeru,     Cyrillic_YERU,	Ukrainian_i,	Ukrainian_I] };
    key <AC03> { [       Cyrillic_ve,       Cyrillic_VE  ] };
    key <AC04> { [        Cyrillic_a,        Cyrillic_A  ] };
    key <AC05> { [       Cyrillic_pe,       Cyrillic_PE  ] };
    key <AC06> { [       Cyrillic_er,       Cyrillic_ER  ] };
    key <AC07> { [        Cyrillic_o,        Cyrillic_O  ] };
    key <AC08> { [       Cyrillic_el,       Cyrillic_EL  ] };
    key <AC09> { [       Cyrillic_de,       Cyrillic_DE  ] };
    key <AC10> { [      Cyrillic_zhe,      Cyrillic_ZHE  ] };
    key <AC11> { [        Cyrillic_e,        Cyrillic_E,	Ukrainian_ie,	Ukrainian_IE] };

    key <AB01> { [       Cyrillic_ya,       Cyrillic_YA  ] };
    key <AB02> { [      Cyrillic_che,      Cyrillic_CHE  ] };
    key <AB03> { [       Cyrillic_es,       Cyrillic_ES  ] };
    key <AB04> { [       Cyrillic_em,       Cyrillic_EM  ] };
    key <AB05> { [        Cyrillic_i,        Cyrillic_I] };
    key <AB06> { [       Cyrillic_te,       Cyrillic_TE  ] };
    key <AB07> { [ Cyrillic_softsign, Cyrillic_SOFTSIGN  ] };
    key <AB08> { [       Cyrillic_be,       Cyrillic_BE  ] };
    key <AB09> { [       Cyrillic_yu,       Cyrillic_YU  ] };

    include "kpdl(comma)"
};

Parting thoughts

Custom keyboard layouts for the win

Tweaking to your purposes something as fundamental as a keyboard layout is strangely empowering. And adapting to a new layout is like learning a foreign language - if you did it at least once in your life, the next ones are much easier. Especially if it’s small things like moving a key, or just adding more symbols to the existing layout.

Wouldn’t recommend it to everyone, though.

One thing I would recommend to everyone without exception is switching the Ctrl and Caps Lock keys. It can be easily done on any OS, including Linux, where too it can be done without editing any layout files¹².

Interesting resources on topic

• X keyboard extension - ArchWiki (of course)
• Custom Keyboard in Linux/X11 | Daniel Paul O’Donnell (“As an early medieval English philologist, I make daily use of a number of characters that are not found on the standard US computer keyboard. These include characters and accented letters from various European languages (e.g. ß, é, ä, ø), IPA symbols (e.g. ŋ, ә, etc.), Middle English ȝ, and various letters and forms used in Old English (æ, þ, ċ, ġ, ƿ, ū, ȳ, etc.).") - dated but very cool
• tonyaldon/keyboard-layout: keyboard-layout pools all the needed files to set up my custom XKB keyboard layout (takbl) on Linux Ubuntu.
• A simple, humble but comprehensive guide to XKB for linux | by Damiano Venturin | Medium

Thank you for reading!