I found out about uv self upgrade which took me from 0.5 to 0.8, read the CHANGELOGs etc. and many neat things exist

Copied from the old Semantic Wiki. Half of the old links are gone, yet another reminder of bit rot and memento mori.

• How to Destroy Performance Anxiety and Deliver an Award-Winning Show Every Time
• How to give a technical presentation (how to give a scientific talk)
• How to give a quals practice talk
• Speaking Resources - Matthew Turland
• public_speaking_for_academic_economists.pdf

Long overdue, will update this page as I find better options.

Tutorials

• Transformer Math 101 | EleutherAI Blog AWESOME — TODO

Links

• LLaMA 7B GPU Memory Requirement - 🤗Transformers - Hugging Face Forums
  • 7B full precision => $7*4=28gb$ of GPU RAM
    • quantization: torch_dtype=torch.float16 etc. to use half the memory
    • this is for inference, training requires a bit more.
  • Why *4? storing weights+gradient, better explanation at that link.
  • based on optimizer, might be *8 etc.
• HF: GPU, more about training but less about mental models

Calculators

• Model Memory Utility - a Hugging Face Space by hf-accelerate
  • manually select ’transformers'
  • results for llama-2.3-1B are 4.6GB total size, 18gb for training
    • doesn’t match info above, but matches e.g. Hardware requirements for Llama 3.2 3B with full context 128k? : r/LocalLLaMA
• LLM Model VRAM Calculator - a Hugging Face Space by NyxKrage

Since forever I have some variations of this:

function lq
    command jq . "$argv" -C | less -r
end

jless - A Command-Line JSON Viewer is this, but much better. Intuitive CLI viewer for json data, works seamlessly for large JSONs (which sometimes breaks my lq command, especially for large lines)

Now:

function lq
	echo "not again, Serhii!!! Use jless/jl"
    command jless -r -m line "$argv"
    # command jq . "$argv" -C | less -r
end

Jless

• m toggles the mode between line-mode (jq-like) and the (default) data mode
• -r does relative numbers! -N disables any line numbers
• h opens the help, the usual vim things do what I expect
  • H focuses the parent

It does yaml as well!

Old langchain code for generating pairs of questions about feminitive usage, didn’t use it in my thesis but don’t want to lose it

from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
from langchain.output_parsers import PydanticOutputParser
from langchain.prompts import PromptTemplate
from langchain.pydantic_v1 import BaseModel, Field, validator
from langchain.schema import HumanMessage
from langchain.prompts import PromptTemplate
from langchain.prompts import ChatPromptTemplate
from langchain.schema import BaseOutputParser
from langchain.output_parsers.json import SimpleJsonOutputParser

from tqdm import tqdm

#  from json import loads
from typing import List

from rich import print, inspect

b = breakpoint

# https://openai.com/pricing?ref=ghostcms.tenten.co
MODEL = "text-ada-001"  # cheap, bad
MODEL = "text-davinci-003"  # acceptable
MODEL = "gpt-4"
MODEL = "gpt-3.5-turbo-1106"  # 'capable, cost-effective'

WOMEN_VARIANTS: str = "дівчина, моя сестра, моя жінка, колишня однокласниця, дочка маминої подруги, імена (Марія, Марія Петрівна, Кассандра, та ін.)"
COMPLETE_PROMPT: str = """Наведи будь-ласка {N_PROFS} однозначні короткі дефініції цій професії або слову, так, щоб по ним було однозначно очевидно про яку саме професію йде мова.

Зміни дефініції так, щоб вони стали фразами, де мова однозначно йде про жінку.  Придумай різні варіанти жінок, про яких йде мова, умовно: {WOMEN_VARIANTS}. Але придумай і свої різноманітніші приклади. 

Уникай використання самого слова чи поняття у визначеннях. Уникай слів 'фахівецька' чи 'спеціалістка'.

Наприклад:
Актор: "Моя жінка виконує ролі на сцені чи екрані" 
Акушерка: "Марія Петрівна допомагає при пологах"
Автор: "Я знаю дівчину, яка пише твори та книжки". 

Будь творчим. Але професія, про яку іде мова, має все рівно бути однозначно зрозумілою.
"""

FORMAT_INSTRUCTIONS = """
Формат виводу - JSON. Обʼєкт виглядати таким чином:
{
    "profession": "", 
    "description_f": ["", ..., ""] 
}

В полі description_f список всіх згенерованих дефініцій для цієї професії.

Виводь тільки код JSON, без ніяких додаткових даних до чи після.
"""

INSTRUCTIONS_GENDER_CHANGE = """Я писатиму речення про професію про жінку. Зміни 
речення так, щоб мова йшла про чоловіка, а не жінку, не міняючи сам опис професії.
Імʼя чи опис жінки можеш міняти як завгодно, головне щоб на виході було речення
про чоловіка. """

def get_model(model_name = None):
    model = OpenAI(model_name=model_name, temperature=0.0)
    return model

def run_and_parse(model, profession: str, n_profs: int | str = 3, women: str = WOMEN_VARIANTS):
    prompt = PromptTemplate(
        template="{complete_prompt}\n{format_instructions}\n Професія, яку потрібно описати: {query}\n",
        input_variables=["query"],
        partial_variables={
            "format_instructions": FORMAT_INSTRUCTIONS,
            "complete_prompt": COMPLETE_PROMPT.format(
                N_PROFS=n_profs, WOMEN_VARIANTS=women
            ),
        },
    )

    json_parser = SimpleJsonOutputParser()
    #  prompt_and_model = prompt | model | json_parser
    prompt_and_model = prompt | model
    model_output = prompt_and_model.invoke({"query": profession})
    output = json_parser.parse(model_output)
    return output

def run_and_parse_gender_change(model, profession_description: str):
    prompt = PromptTemplate(
        template="{complete_prompt}\n Речення наступне: {query}\n",
        input_variables=["query"],
        partial_variables={
            #  "format_instructions": FORMAT_INSTRUCTIONS,
            "complete_prompt": INSTRUCTIONS_GENDER_CHANGE
            },
    )

    #  json_parser = SimpleJsonOutputParser()
    #  prompt_and_model = prompt | model | json_parser
    prompt_and_model = prompt | model
    model_output = prompt_and_model.invoke({"query": profession_description})
    output =model_output
    #  b()
    #  output = json_parser.parse(model_output)
    return output

def generate_descriptions(model, profession: str, n_profs: int | str = 3, women: str = WOMEN_VARIANTS, do_male_version:bool = False):
    desc = run_and_parse(model=model, profession=profession, n_profs =n_profs, women=women)
    if do_male_version:
        description_male = list()
        for d in desc['description_f']:
            changed = run_and_parse_gender_change(model=model, profession_description=d)
            description_male.append(changed)
        desc['description_m'] = description_male
    return desc


def run():
    professions_raw = """
    абстракціоністка
    автомобілістка
    авторка
    """
    """
    агрономка
    адвокатка
    анархіст
    англієць
    антрополог
    асистентка
    астронавт
    аптекар
    """
    profs = [x.strip() for x in professions_raw.splitlines()]

    model=get_model(MODEL)

    results = list()
    for p in tqdm(profs):
        r = generate_descriptions(model=model, profession=p, n_profs=2)
        print(r)
        results.append(r)
    print(results)


if __name__ == "__main__":
    run()

    {
        'profession': 'лікар',
        'description_f': [
            'Моя сестра працює в лікарні та лікує хворих',
            'Дочка маминої подруги є лікарем та допомагає людям'
        ]
    },
    {
        'profession': 'абстракціоністка',
        'description_f': ['Моя сестра створює картини, які відображають абстрактні ідеї та почуття',
        'Дівчина, яку я знаю, малює абстракціоністські полотна'
        ]
    },    {
        'profession': 'автомобілістка',
        'description_f': [
            'Моя сестра вміє водити автомобіль',
            'Дочка маминої подруги працює водієм'
        ]
    },
    {
        'profession': 'авторка',
        'description_f': [
            'Моя сестра пише книги та статті',
            'Дочка маминої подруги є відомою письменницею'
        ]
    },
    {
        'profession': 'Вчитель',
        'description_f': [
            'Моя сестра працює в школі та навчає дітей',
            'Дочка маминої подруги викладає університетські предмети'
        ]
    }
]

• [2103.05331] Active Testing: Sample-Efficient Model Evaluation
  • <_(@kossenactivetesting2021) “Active Testing: Sample-Efficient Model Evaluation” (2021) / Jannik Kossen, Sebastian Farquhar, Yarin Gal, Tom Rainforth: z / http://arxiv.org/abs/2103.05331 / 10.48550/arXiv.2103.05331 _>¹
  • Introduces active testing
• [2101.11665] On Statistical Bias In Active Learning: How and When To Fix It the paper describing the estimator for AL in detail, the one that removes bias when sampling
• [2508.09093] Scaling Up Active Testing to Large Language Models
  • <_(@berradascalingactive2025) “Scaling Up Active Testing to Large Language Models” (2025) / Gabrielle Berrada, Jannik Kossen, Muhammed Razzak, Freddie Bickford Smith, Yarin Gal, Tom Rainforth: z / http://arxiv.org/abs/2508.09093 / 10.48550/arXiv.2508.09093 _> ²
  • adapts it to LLMs to LLMs

OK, so.

Active testing¹

• Active learning: picking the most useful training instances to the model (e.g. because annotation is expensive, and we pick what to annotate)

• Active testing: pick the most useful subset of testing instances that approximates the score of the model on the full testing set.

• We can test(=I use label below) only a subset of the full test set, $D_{test}^{observed}$ from $D_{test}$

• We decide to label only a subset $N>M$ of the test instances, and not all together, but one at a time, because then we can use the information of the already labeled test instances to pick which next one to label

• It calls these (estimated or not) test scores test risk, $R$ .

Strategies

• Naive: we uniformly sample the full test dataset
  • BUT if $M«N$, its variance will be large, so though it’s uniformly/unbiasedly sampled, it won’t necessarily approximate the real score well
• Actively sampling, to reduce the variance of the estimator.
  • Actively selecting introduces biases as well if not done right
    • e.g. active learning you label hard points to make it informative, but in active testing testing on hard points will inflate the risk

Active sampling

• sth monte carlo etc. etc.
• TODO grok the math and stats behind all of this.
  • follows my understanding that might be wrong
  • The [2101.11665] On Statistical Bias In Active Learning: How and When To Fix It paper that introduces $R_{lure}$ is might be a starting point but at first sight it’s math-heavy
• TL;DR pick the most useful points, and get rid of the biases this introduces (e.g. what if they are all complex?) by using smart auto-correcting mechanisms (=‘unbiased estimator’)
• You have some $q(i_m)$ strategy that picks interesting samples, though they might bias the result.
  • $q(i_m)$ is actually shorthand for $q(i_m, i_{1:m-1}, D_{test}, D_{train})$ — so we have access to the the datasets and the already aquired test data.
• $R_{LURE}$: An unbiased estimator that auto-corrects the discribution to be closer to the one expected if you iid-uniformly sample the test dataset.
• ChatGPT on the intuition behind this:
  • also:

  This is the classic promise of importance sampling: move probability mass toward where the integrand is large/volatile, then down-weight appropriately.

  • also:

  Bottom line: active testing = (i) pick informative test points stochastically with $q$; (ii) compute a weighted Monte Carlo estimate with $v_m​$ to remove selection bias; (iii) enjoy lower variance if $q$ is well-chosen.

• argparse - parse options passed to a fish script or function — fish-shell 4.0.2 documentation
• function - create a function — fish-shell 4.0.2 documentation

Below:

• h/help is a flag
• o/output-dir gets+requires a value
• or return fails on unknown -a/--args
• If found, the value is removed from $argv and put into _flag_argname
• set -query (tests if var is set) -local

argparse h/help o/output-dir= -- $argv 
or return

if set -ql _flag_help
	echo "Usage: script.fish [-o/--output-dir <dir>] <input_files_or_directory>"
	echo "Example: script.fish --output-dir /path/to/output file1.png directory_with_pngs/ dir/*.png"
	exit 0
end

if set -ql _flag_output_dir
	set output_dir $_flag_output_dir
end


for arg in $argv
	# do something with file $arg
end

Scenario:

• your uv source package (that lives in a gitlab source project) depends on a target package from a gitlab target project’s package registry.
• you want uv add etc. to work transparently
• you want gitlab CI/CD pipelines to work transparently

Links

• Package indexes | uv
• Authentication | uv
• PyPI packages in the package registry | GitLab Docs

TL;DR

• Have a gitlab token with at least read_api scope and Developer+ role
• Find the address of the registry through Gitlab UI
  • https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple
  • https://gitlab.de/api/v4/projects/1111/packages/pypi/simple
  • FROM A GROUP¹: https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi/simple
  • ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi
• pyproject: add it registry, see below
• to authenticate UV, you can use:
  • token inside URI, or
  • UV_INDEX_PRIVATE_REGISTRY_USERNAME/PASSWORD env variables, replacing PRIVATE_REGISTRY with the name you gave to it in pyproject.toml
  • ~/.netrc file: .netrc - everything curl

Gitlab

• In Gitlab, use/create an access token with read_xxx permissions in the project — read_api, read_repository, read_registry are enough — and Developer role.
• To get the package registry address,
  • Use this URI for the group registry (it’s the best for multiple projects in the same group): https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi/simple
    • You can get the group ID from the group page, hamburger menu in upper-right, “copy group id”. Will be an int.
  • open the registry, click on a package, click “install” and you’ll see it all:
    • different packages in different projects will have different registries!
  • The project repository URI is generally ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi/simple

UV

Add the new package registry as index

tool.uv.index
name = "my-registry"
url = "https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple"
# authenticate = "always" # see below
# ignore-error-codes = [401]

The URI either contains token inside URI or doesn’t. The examples below are /projects/xxx, ofc a group registry works as well. - https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple - token inside the URI - https://gitlab.de/api/v4/projects/1111/packages/pypi/simple — auth happening through env. variables or ~/.netrc

Uv auth

Through the server URI

url = "https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple"

Through env variables:

export UV_INDEX_PRIVATE_REGISTRY_USERNAME=__token__ 
export UV_INDEX_PRIVATE_REGISTRY_PASSWORD=glpat-secret-token

PRIVATE_REGISTRY needs to be replaced with the name of the registry . So e.g. for the pyproject above it’s UV_INDEX_**MY_REGISTRY**_USERNAME.

Through a .netrc file

From Authentication | uv / HTTP Authentication and PyPI packages in the package registry | GitLab Docs: Create a ~/.netrc:

machine gitlab.example.com
login __token__
password <personal_token>

It will use these details, when you uv add ... -v you’d see a line like

DEBUG Checking netrc for credentials for https://gitlab.de/api/v4/projects/1111/packages/pypi/simple/packagename/
DEBUG Found credentials in netrc file for https://gitlab.de/api/v4/projects/1111/packages/pypi/simple/packagename/

NB git will also use these credentials — so if the token’s scope doesn’t allow e.g. pushing, you won’t be able to git push. Use a wider scope or a personal access token (or env. variables)

Usage

• When you uv add yourpackage, uv looks for packages in all registries
  • The usual pypi one is on by default
• If gitlab doesn’t find one in the gitlab registry or the user is unauthenticated, gitlab by default transparently mirrors pypi
• If auth fails for any of the indexes, uv will fail loudly — you can ignore-error-codes = [401] to make uv keep looking inside the other registries

Gitlab CI/CD pipelines

CI/CD pipelines have to have access to the package as well, when they run.

GitLab CI/CD job token | GitLab Docs:

You can use a job token to authenticate with GitLab to access another group or project’s resources (the target project). By default, the job token’s group or project must be added to the target project’s allowlist.

In the target project (the one that needs to be resolved, the one with the private registry), in Settings->CI/CD -> Job token permissions add the source project (the one that will access the packages during CI/CD).

You can just add the group parent of all projects as well, then you don’t have to add any individual ones.

Then $CI_JOB_TOKEN can be used to access the target projects. For example, through a ~/.netrc file (note the username!)

machine gitlab.example.com
username gitlab-ci-token
password $CI_JOB_TOKEN

Bonus round: gitlab-ci-local

I love firecow/gitlab-ci-local.

When running gitlab-ci-local things, the CI_JOB_TOKEN variable is empty. You can create a .gitlab-ci-local-variables.yaml (don’t forget to gitignore it!) with this variable, it’ll get used automatically and your local CI/CD pipelines will run as well:

CI_JOB_TOKEN=glpat-secret-token

Tutorial and bird’s eye view - Jujutsu docs Git replacement using git under the hood.

First encountered here: Jujutsu for busy devs | Hacker News / Jujutsu For Busy Devs | maddie, wtf?!

For anyone who’s debating whether or not jj is worth learning, I just want to highlight something. Whenever it comes up on Hacker News, there are generally two camps of people: those who haven’t given it a shot yet and those who evangelize it.

Alright, let’s try!

To use on top of an existing git repository:

jj git init --colocate .
jj git clone --colocate git@github.com:maddiemort/maddie-wtf.git

Command line completions: COMPLETE=fish jj | source

How to Firefox | Hacker News / 🦊 How to Firefox - Kaushik Gopal’s Website

• Type / and start typing for quick find (vs ⌘F). But dig this, ' and Firefox will only match text for hyper links
• URL bar search shortcuts: * for bookmarks, % for open tabs, ^ for history
• If you have an obnoxious site disable right click, just hold Shift and Firefox will bypass and show it to you. No add-one required.

(emph mine)

Damn. DAMN.

I need to set this up in qutebrowser as well, it’s brilliant.

For later: Introduction | LLM Inference in Production

autorandr -c vertical-reverse describes my home layout, autorandr -c horizontal describes my work layout. Awesome.

 The following virtual configurations are available:
  off             Disable all outputs
  common          Clone all connected outputs at the largest common resolution
  clone-largest   Clone all connected outputs with the largest resolution (sca-
                  led down if necessary)
  horizontal      Stack all connected outputs horizontally at their largest re-
                  solution
  vertical        Stack all connected outputs vertically at their largest reso-
                  lution
  horizontal-reverse Stack all connected outputs horizontally at their largest re-
                  solution in reverse order
  vertical-reverse Stack all connected outputs vertically at their largest reso-
                  lution in reverse order

Previously:

• more generic: 250317-1847 Current LLM evaluation landscape
• 250505-1019 Evaluating RAG
• 250415-1640 Evaluating NLG

EleutherAI lm-eval-harness

Link: EleutherAI/lm-evaluation-harness: A framework for few-shot evaluation of language models.

Running

Running HF model with model args (hf model name in model_args as well):

lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size 8

Task formats

YAML+jinja, can run python code in some of the params.

task: coqa
dataset_path: EleutherAI/coqa
output_type: generate_until
training_split: train
validation_split: validation
doc_to_text: !function utils.doc_to_text
doc_to_target: !function utils.doc_to_target
process_results: !function utils.process_results
should_decontaminate: true
doc_to_decontamination_query: "{{story}} {{question.input_text|join('\n')}}"
generation_kwargs:
  until:
    - "\nQ:"
metric_list:
  - metric: em
    aggregation: mean
    higher_is_better: true
  - metric: f1
    aggregation: mean
    higher_is_better: true

Features

• Supports HF models/datasets
• Jinja templates, separate packages for IFeval, promptsource etc.
• Many options for post-processing, answer extraction, fewshot etc. the usual, all live in the YAML

Inference

• HF, incl. local, OpenAI API format, but other frameworks as well — nvidia nemo etc.
• Llama.cpp / vLLM
• multi-GPU with accelerate: accelerate launch -m lm_eval --model ...

Saving and caching

• JSON through --output_path param
• --log_samples logs samples
• --use-cache caches stuff and reruns it only when needed
• --hf_hub_log_args logs the results to HF ! (documentation broken though)

Misc

• Last (most complex) version of HF Hub leaderboard (before it got discontinued): lm-evaluation-harness/lm_eval/tasks/leaderboard/README.md at main · EleutherAI/lm-evaluation-harness
• Python integration: simple_evaluate(): lm-evaluation-harness/docs/interface.md at main · EleutherAI/lm-evaluation-harness

lmms-eval

• EvolvingLMMs-Lab/lmms-eval: Accelerating the development of large multimodal models (LMMs) with one-click evaluation module - lmms-eval.
• very close in spirit to eval-harness but multimodal

Sample¹:

export HF_HOME="~/.cache/huggingface"
export AZURE_OPENAI_API_KEY=""
export AZURE_OPENAI_API_BASE=""
export AZURE_OPENAI_API_VERSION="2023-07-01-preview"

# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
python3 -m lmms_eval \
    --model openai_compatible \
    --model_args model_version=gpt-4o-2024-11-20,azure_openai=True \
    --tasks mme,mmmu_val \
    --batch_size 1

Task yamls look very similar: lmms-eval/lmms_eval/tasks/gqa/gqa.yaml at main · EvolvingLMMs-Lab/lmms-eval

Evalverse

• Evalverse | Evalverse Docs
• Looks abandoned
• One package to run multiple eval frameworks:
  • lm-eval for usual benchmarks
  • Chat: evaluated using FastChat llm-as-a-judge
    • MT-Bench
    • IFEval
    • EQ-Bench
• Their params:Evaluation | Evalverse Docs
  • also “Common Arguments” (hah)
  • And then arguments for each benchmark separately

Architecture

Evaluator runs the library given by the Connector

Running

(h6_en is lm-eval)

Results

• JSON, one for each task.
• They have a Reporter class that summarizes and analyzes the results saved by the evaluator

OpenAI evals

• Link: openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
• I like their documentation!

Running

oaieval gpt-3.5-turbo test-match

Task formats

Data

Data has to be in JSONL format.

{
  "input": [
    {
      "role": "system",
      "content": "You are an assistant with knowledge of U.S. state laws. Answer the questions accurately."
    },
    {
      "role": "user",
      "content": "List the states where adultery is technically illegal. Only provide a list of states with no explanation."
    }
  ],
  "ideal": "Alabama, Arizona, Florida, Idaho, Illinois, Kansas, Michigan, Minnesota, Mississippi, New York, North Carolina, Oklahoma, Rhode Island, South Carolina, Virginia, Wisconsin, Georgia"
}

Building an eval

Registering the eval²:

<eval_name>:
  id: <eval_name>.dev.v0
  description: <description>
  metrics: [accuracy]

<eval_name>.dev.v0:
  class: evals.elsuite.basic.match:Match
  args:
    samples_jsonl: <eval_name>/samples.jsonl

Eval templates

• evals/docs/eval-templates.md at main · openai/evals templates to make task creation easier
  • e.g. basic: match/fuzzy-match/..
  • model-graded: use a model to parse the completion into an easily parsable shape, e.g. yes/no/maybe.

Model-graded templates

Sample yaml:³

humor_likert:
  prompt: |-
    Is the following funny?

    {completion}

    Answer using the scale of 1 to 5, where 5 is the funniest.
  choice_strings: "12345"
  choice_scores: from_strings
  input_outputs:
    input: completion

closedqa:            config_dict = yaml.load(yaml_path.read_text())
  prompt: |-
    You are assessing a submitted answer on a given task based on a criterion. Here is the data:
    [BEGIN DATA]
    ***
    [Task]: {input}
    ***
    [Submission]: {completion}
    ***
    [Criterion]: {criteria}
    ***
    [END DATA]
    Does the submission meet the criterion? First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print only the single character "Y" or "N" (without quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the letter again by itself on a new line.

    Reasoning:
  eval_type: cot_classify
  choice_scores:
    "Y": 1.0
    "N": 0.0
  choice_strings: 'YN'
  input_outputs:
    input: "completion"

• Includes
  • closedqa (is the answer given in the prompt?)
  • battle.yaml: pairwise choice basically
• Notably, I don’t really see a way to do multiple choice using their basic eval templates, a la lm-eval doc_to_choice.

Inference

• Completion functions: evals/docs/completion-fns.md at main · openai/evals
• generalization of “completion”, e.g. also to allow the model to open a browser or whatever

langchain/llm/text-davinci-003:
  class: evals.completion_fns.langchain_llm:LangChainLLMCompletionFn
  args:
    llm: OpenAI
    llm_kwargs:
      model_name: text-davinci-003

langchain/llm/flan-t5-xl:
  class: evals.completion_fns.langchain_llm:LangChainLLMCompletionFn
  args:
    llm: HuggingFaceHub
    llm_kwargs:
      repo_id: google/flan-t5-xl

Model types

Not immediately easily exposed, definitely supports OpenAI and LangChain and HF, but it’s not intuitive.

HF Lighteval

Lighteval

Running

lighteval accelerate \
     "model_name=openai-community/gpt2" \
     "leaderboard|truthfulqa:mc|0|0"

The syntax: {suite}|{task}|{num_few_shot}|{0 for strict num_few_shots, or 1 to allow a truncation if context size is too small}

Task formats

• Python files: Adding a Custom Task.
• Example template: lighteval/community_tasks/_template.py
• German RAG eval: lighteval/community_tasks/german_rag_evals.py at main · huggingface/lighteval
• Currently implemented tasks: lighteval/src/lighteval/tasks at main · huggingface/lighteval
• Example task: lighteval/examples/custom_tasks_templates/custom_yourbench_task_mcq.py at main · huggingface/lighteval

    return Doc(
        instruction=ZEROSHOT_QA_INSTRUCTION,
        task_name=task_name,
        query=ZEROSHOT_QA_USER_PROMPT.format(question=line["question"], options=options),
        choices=line["choices"],
        gold_index=gold_index,
    )

yourbench_mcq = LightevalTaskConfig(
    name="HF_TASK_NAME",  # noqa: F821
    suite=["custom"],
    prompt_function=yourbench_prompt,
    hf_repo="HF_DATASET_NAME",  # noqa: F821
    hf_subset="lighteval",
    hf_avail_splits=["train"],
    evaluation_splits=["train"],
    few_shots_split=None,
    few_shots_select=None,
    generation_size=8192,
    metric=[Metrics.yourbench_metrics],
    trust_dataset=True,
    version=0,
)

Features

Inference

Model types

Many, and model configs are yamls: lighteval/examples/model_configs at main · huggingface/lighteval. For example: lighteval/examples/model_configs/litellm_model.yaml at main · huggingface/lighteval

model_parameters:
  model_name: "openai/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
  provider: "openai"
  base_url: "https://router.huggingface.co/hf-inference/v1"
  generation_parameters:
    temperature: 0.5
    max_new_tokens: 256
    top_p: 0.9
    seed: 0
    repetition_penalty: 1.0
    frequency_penalty: 0.0

Misc

Their default prompts: lighteval/src/lighteval/tasks/default_prompts.py at main · huggingface/lighteval

HELM

• stanford-crfm/helm: Holistic Evaluation of Language Models (HELM)
• Their famous leaderboard: Holistic Evaluation of Language Models (HELM)
• Documentation: Index - CRFM HELM

Running

# Run benchmark 
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10 
# Summarize benchmark results 
helm-summarize --suite my-suite

Task formats

Features

Inference

A model has metadata (description) and deployment (actually how to run it / implementation). Adding New Models - CRFM HELM Both are yamls.

HF model deployiment (but running locally!):

  - name: huggingface/gemma-2-9b-it
    model_name: google/gemma-2-9b-it
    tokenizer_name: google/gemma-2-9b
    max_sequence_length: 8192
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        device_map: auto
        torch_dtype: torch.bfloat16

I’m not certain how that connects to their Hugging Face Model Hub Integration - CRFM HELM — tl;dr only AutoModelForCausalLM. To run: helm-run \ --run-entries boolq:model=stanford-crfm/BioMedLM \ --enable-huggingface-models stanford-crfm/BioMedLM \ --suite v1 \ --max-eval-instances 10

All (many!) deployments: helm/src/helm/config/model_deployments.yaml at main · stanford-crfm/helm

vLLM example uses a OpenAI-compatible inference server

OpenCompass

• Welcome to OpenCompass’ documentation! — OpenCompass 0.4.2 documentation
• open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

Never heard of it but looks cool! And supports many types of evals.

Subjective Evaluation Guidance — OpenCompass 0.4.2 documentation

All configs in Python! models tasks etc.

Model types

All configs in python!

# model_cfg.py
from opencompass.models import HuggingFaceCausalLM

models = [
    dict(
        type=HuggingFaceCausalLM,
        path='huggyllama/llama-7b',
        model_kwargs=dict(device_map='auto'),
        tokenizer_path='huggyllama/llama-7b',
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
        max_seq_len=2048,
        max_out_len=50,
        run_cfg=dict(num_gpus=8, num_procs=1),
    )
]

OpenAI:

from opencompass.models import OpenAI

models = [
    dict(
        type=OpenAI,                             # Using the OpenAI model
        # Parameters for `OpenAI` initialization
        path='gpt-4',                            # Specify the model type
        key='YOUR_OPENAI_KEY',                   # OpenAI API Key
        max_seq_len=2048,                        # The max input number of tokens
        # Common parameters shared by various models, not specific to `OpenAI` initialization.
        abbr='GPT-4',                            # Model abbreviation used for result display.
        max_out_len=512,                         # Maximum number of generated tokens.
        batch_size=1,                            # The size of a batch during inference.
        run_cfg=dict(num_gpus=0),                # Resource requirements (no GPU needed)
    ),
]

OpenCompass VLMEvalKit

Github: open-compass/VLMEvalKit: Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Same creators as the above one, multimodal eval.

DeepEval

• confident-ai/deepeval: The LLM Evaluation Framework
• basically unit-test for LLMs — optionally literally using pytest!
• “large amount of metrics” usable with “ANY LLM of your choice”
  • includes RAG tool use etc.
• have their free cloud thingy to share reports etc., optional but they push it heavily. Something like wandb — visualize scores compare etc.
• Supports evaluating LlamaIndex | DeepEval

Tasks

From their README:

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        # Replace this with the actual output from your LLM application
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
    )
    assert_test(test_case, [correctness_metric])

Models

• the usual ones, including neat HF support k The vLLM bit describes using an openAI-compatible endpoints: vLLM | DeepEval - The Open-Source LLM Evaluation Framework, same as for LMStudio

deepeval set-local-model --model-name=<model_name> \  
--base-url="http://localhost:8000/v1/" \  
--api-key=<api-key>

Misc

• Can do component-based evals through @observe decorators, “avoiding rewriting your app just for testing”
• Supports many vector databases eval for RAG retrieval eval

Other stuff

FastChat MTBench

• FastChat/fastchat/llm_judge at main · lm-sys/FastChat
• pure Python, llm-as-a-judge
• one python script to generate answers, another to get LLM judgements, a third to calculate results and draw plots — neat pattern
• FastChat powers LLM Arena, and also provides an OpenAI-compatible API for the models it supports

Autoevals

• braintrustdata/autoevals: AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.
• python+TS, no runner but framework
• API-first, doesn’t try to run local inference, which again I like
• Generally I really like it

Pydantic evals

• Evals - PydanticAI
• Yet another unit tests for LLM approach, flexible, supports LLM-as-a-judge

https://www.keybr.com/ looks really cool!

Just some really quick notes on this, it’s pointless and redundant but I’ll need these later

• lm-sys/FastChat: An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
  • FastChat/docs/model_support.md at main · lm-sys/FastChat
• LiteLLM Proxy Server (LLM Gateway) | liteLLM

LiteLLM

Create .yaml with a github models model:

model_list:
  # - model_name: github-Llama-3.2-11B-Vision-Instruct # Model Alias to use for requests
  - model_name: minist # Model Alias to use for requests
    litellm_params:
      model: github/Ministral-3B
      api_key: "os.environ/GITHUB_API_KEY" # ensure you have `GITHUB_API_KEY` in your .env

After setting GITHUB_API_KEY, litellm --config config.yaml

FastChat

• Install FastChat, start the controller:
  • python3 -m fastchat.serve.controller
• FastChat model config:

{
    "minist": {
        "model_name": "minist",
        "api_base": "http://0.0.0.0:4000/v1",
        "api_type": "openai",
        "api_key": "whatever",
        "anony_only": false
    }
}

• Run the webUI
  • python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file ../model_config.json
  • Direct chat now works!

TIL about 1C when I had to move it from one windows laptop to another. First and last windows post here hopefully

Ref: 1С – как перенести базу на другой компьютер

Long story short:

• 1C v 7.7 from 1999 can be literally copied to another laptop and it works, windows is good at this.
• the database is also a directory, which is usually shown when you start 1C and choose the DB to open
• that directory can be copied
  • I THINK it’s good karma if the path on both computers is identical, but am not certain
• if the version matches that should be it
• no idea how to check if it worked :)

Kitty was really slow to start and this has been bugging me. Especially setting up a new system, using the default gnome-terminal, and seeing it appear instantly.

Kitty’s single instance mode (also -1) decreased start from 400ms to 300, still too much.

time alacritty -e bash -c exit
time gnome-terminal -e "bash -c exit"
time kitty --single-instance bash -c exit

New best friend: Alacritty

Saw Alacritty mentioned and it’s awesome. Has everything I wanted from or set up for kitty. Kitty is more configurable (I think), but I’m not missing anything at all so far.

Spawn instance in same directory

I used to have a separate command for that!

ArchWiki:

[keyboard]
bindings = [
   { key = "Return", mods = "Control|Shift", action = "SpawnNewInstance" }
]

Hints

Updated the default config to copy instead of launch, use better letters, and do file paths together with URIs:

[hints]
alphabet = "aoeusndh"

hints.enabled
action = "Copy"
# command = "xdg-open" # On Linux/BSD
hyperlinks = true
post_processing = true
persist = false
mouse.enabled = true
binding = { key = "N", mods = "Control|Shift" }

# adds file paths as well
regex = '(?:(?:ipfs:|ipns:|magnet:|mailto:|gemini://|gopher://|https?://|news:|git://|ssh:|ftp:|file:)[^\u0000-\u001F\u007F-\u009F<>"\s{}\-\^⟨⟩`\\]+|(?:(?:\.\.?/)+|/)[^\u0000-\u001F\u007F-\u009F<>"\s{}\-\^⟨⟩`\\]+)'
# regex = "(ipfs:|ipns:|magnet:|mailto:|gemini://|gopher://|https://|http://|news:|file:|git://|ssh:|ftp://)[^\u0000-\u001F\u007F-\u009F<>\"\\s{-}\\^⟨⟩`\\\\]+"

Vim mode

<C-S-Space> runs a vim-ish mode on the text, one can then copy etc. with all usual movements!

In kitty I had to do vim as scrollback pager etc. (old code below, prolly broken, didn’t use it because too complex)

For later

• alacritty/alacritty-theme: Collection of Alacritty color schemes

Previously

Some old configs from kitty, for reference:

## Bindings

# https://sw.kovidgoyal.net/kitty/index.html#kittens
map kitty_mod>f1 launch --stdin-source=@screen_scrollback --stdin-add-formatting less +G -R

## Hints
# File paths
map kitty_mod+n>f kitten hints --type path --program @

# IPs (+ with ports)
map kitty_mod+n>i kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3} --program @
map kitty_mod+n>p kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3}:[0-9]+ --program @

# CLI Commands
# map kitty_mod+n>c kitten hints --type regex --regex "(\$|>)(.+)(?:\n|\s*$)?" --program @
# This version copies up to the vim mode indicator
# map kitty_mod+n>c kitten hints --type regex --regex "(\$|>)(.+?)(?:\n|\s+$|\s+(?:INS|VIS|REP|SEA))" --program @
# map kitty_mod+n>c kitten hints --type regex --regex "\$(.+)" --program @

# Linenum
map kitty_mod+n>l kitten hints --type line --program @

Scrollback/vim:

# https://sw.kovidgoyal.net/kitty/index.html#kittens
map kitty_mod>f1 launch --stdin-source=@screen_scrollback --stdin-add-formatting less +G -R
scrollback_pager vim - -c "w! /tmp/kitty_scrollback_sh" -c "term ++curwin cat /tmp/kitty_scrollback_sh"

• After running a command (e.g. ag whatever, ^ag^rg re-runs it substituting the first part with the second — so rg whatever)
  • Bash History: Correct / Repeat The Last Command With a Substitution - nixCraft
• Fish doesn’t (want to) support this: Frequently asked questions — fish-shell 4.0.1 documentation

Ty BB for this

command line - How to do: underline, bold, italic, strikethrough, color, background, and size in Gnome Terminal? - Ask Ubuntu:

echo -e '\e[1mbold\e[22m'
echo -e '\e[2mdim\e[22m'
echo -e '\e[3mitalic\e[23m'
echo -e '\e[4munderline\e[24m'
echo -e '\e[4:1mthis is also underline (since 0.52)\e[4:0m'
echo -e '\e[21mdouble underline (since 0.52)\e[24m'
echo -e '\e[4:2mthis is also double underline (since 0.52)\e[4:0m'
echo -e '\e[4:3mcurly underline (since 0.52)\e[4:0m'
echo -e '\e[4:4mdotted underline (since 0.76)\e[4:0m'
echo -e '\e[4:5mdashed underline (since 0.76)\e[4:0m'
echo -e '\e[5mblink (since 0.52)\e[25m'
echo -e '\e[7mreverse\e[27m'
echo -e '\e[8minvisible\e[28m <- invisible (but copy-pasteable)'
echo -e '\e[9mstrikethrough\e[29m'
echo -e '\e[53moverline (since 0.52)\e[55m'

echo -e '\e[31mred\e[39m'
echo -e '\e[91mbright red\e[39m'
echo -e '\e[38:5:42m256-color, de jure standard (ITU-T T.416)\e[39m'
echo -e '\e[38;5;42m256-color, de facto standard (commonly used)\e[39m'
echo -e '\e[38:2::240:143:104mtruecolor, de jure standard (ITU-T T.416) (since 0.52)\e[39m'
echo -e '\e[38:2:240:143:104mtruecolor, rarely used incorrect format (might be removed at some point)\e[39m'
echo -e '\e[38;2;240;143;104mtruecolor, de facto standard (commonly used)\e[39m'

echo -e '\e[46mcyan background\e[49m'
echo -e '\e[106mbright cyan background\e[49m'
echo -e '\e[48:5:42m256-color background, de jure standard (ITU-T T.416)\e[49m'
echo -e '\e[48;5;42m256-color background, de facto standard (commonly used)\e[49m'
echo -e '\e[48:2::240:143:104mtruecolor background, de jure standard (ITU-T T.416) (since 0.52)\e[49m'
echo -e '\e[48:2:240:143:104mtruecolor background, rarely used incorrect format (might be removed at some point)\e[49m'
echo -e '\e[48;2;240;143;104mtruecolor background, de facto standard (commonly used)\e[49m'

echo -e '\e[21m\e[58:5:42m256-color underline (since 0.52)\e[59m\e[24m'
echo -e '\e[21m\e[58;5;42m256-color underline (since 0.52)\e[59m\e[24m'
echo -e '\e[4:3m\e[58:2::240:143:104mtruecolor underline (since 0.52) (*)\e[59m\e[4:0m'
echo -e '\e[4:3m\e[58:2:240:143:104mtruecolor underline (since 0.52) (might be removed at some point) (*)\e[59m\e[4:0m'
echo -e '\e[4:3m\e[58;2;240;143;104mtruecolor underline (since 0.52) (*)\e[59m\e[4:0m'

Getting Started | Task

Jan-Piet Mens :: A shell command to create JSON: jo

• Lua-guide - Neovim docs
  • tl;dr vim.opt.xxx = yy is the most conv. way to set all local global options.
  • vim.wo etc. exist for easier access

vim.wo.relativenumber = true
vim.wo.cursorcolumn = true
vim.wo.cursorline = true
vim.wo.colorcolumn = "80"

-- :set tabstop=8 shiftwidth=4 softtabstop=4 expandtab shiftround
vim.g.tabstop = "8"
vim.g.softtabstop = "8"
vim.g.shiftwidth = "4"
vim.g.expandtab = true
vim.g.smarttab = true

General info

• Roughly, it’s evaluating
  • context — how relevant/correct are the retrieved chunks
  • answer — how good are the generated claims
  • (with interplay inbetween — e.g. whether the answer comes from the context, regardless of whether both are relevant to the query)
• Or: generator metrics, retriever metrics, and overall metrics as used in RAGChecker (see RAGChecker picture later below).

Sources / libs

• Evaluating - LlamaIndex

  • Especially Usage Pattern (Response Evaluation) - LlamaIndex
    • Faithfullness: context(=source)+answer => whether the answer comes from the context available (=whether the answer was hallucinated)
    • Relevancy: query+context+answer => whether the answer AND CONTEXT were relevant for the specific query
  • All LlamaIndex’s eval modules: Modules - LlamaIndex
    • CorrectnessEvaluator: compare query+answer to a reference answer: Correctness Evaluator - LlamaIndex w/ another LLM
    • Embedding Similarity Evaluator - LlamaIndex response+reference semantic similarity via embeddings

• Deepeval’s metrics as given in the llamaindex docs:

  from deepeval.integrations.llama_index import ( DeepEvalAnswerRelevancyEvaluator, DeepEvalFaithfulnessEvaluator, DeepEvalContextualRelevancyEvaluator, DeepEvalSummarizationEvaluator, DeepEvalBiasEvaluator, DeepEvalToxicityEvaluator, )
  

RAGChecker

• RAGChecker: RAGChecker: A Fine-grained Framework For Diagnosing RAG
  • LlamaIndex integration docs: RAGChecker: A Fine-grained Evaluation Framework For Diagnosing RAG - LlamaIndex

• https://blog.edward-li.com/tech/advanced-python-features/
  • Protocols
  • Python slots to make accessing fields of a class faster
  • for-else statements to avoid if x is not None(): ...
  • etc., TODO a lot of cool stuff
• https://news.ycombinator.com/item?id=43770494

* did you know __init__.py is optional nowadays?

* you can do relative imports with things like "from ..other import foo"

* since 3.13 there is a @deprecated decorator that does what you think it does

* the new generics syntax also works on methods/functions: "def method[T](...)" very cool

* you can type kwargs with typeddicts and unpack: "def fn(*kwargs: Unpack[MyKwargs])"

* dataclasses (and pydantic) support immutable objects with: "class MyModel(BaseModel, frozen=True)" or "@dataclass(frozen=True)"

* class attributes on dataclasses, etc. can be defined with "MY_STATIC: ClassVar[int] = 42" this also supports abstract base classes (ABC)

* TypeVar supports binding to enforce subtypes: "TypeVar['T', bound=X]", and also a default since 3.13: "TypeVar['T', bound=X, default=int]"

* @overload is especially useful for get() methods to express that the return can't be none if the default isn't None

* instead of Union[a, b] or Optional[a] you can write "a | b" or "a | None" nowadays

* with match you can use assert_never() to ensure exhaustive matching in a "case _:" block

* typing has reveal_type() which lets mypy print the type it thinks something is

* typing's "Self" allows you to more properly annotate class method return types

* the time package has functions for monotonic clocks and others not just time()

• Selecting and Ignoring Violations — flake8 7.2.0 documentation

  • https://www.flake8rules.com/

• mypy

  • Error codes - mypy 1.15.0 documentation
  • Type inference and type annotations - mypy 1.15.0 documentation

Ignoring files:

# type: ignore
# flake8: noqa
# pylint: skip-file

[]

Autosave and format

    "files.autoSave": "onFocusChange",
    "[python]": {
        "editor.formatOnSave": true,
        // "editor.defaultFormatter": "charliermarsh.ruff",
		"editor.defaultFormatter": "ms-python.black-formatter",
		// reformat everything w/ ruff
        "editor.codeActionsOnSave": {
            "source.fixAll": "explicit",
            "source.organizeImports": "explicit"
        },
    },

Misc

    "editor.rulers": [
        78, 88
    ],
	"editor.lineNumbers": "relative",
	"editor.formatOnPaste": true,
    "editor.formatOnSave": true,  // non-python stuff like settings.json
	

    "debug.allowBreakpointsEverywhere": true,
    "debug.inlineValues": "on",

Vim

<S-j>/<S-k> to switch tabs

    "vim.normalModeKeyBindingsNonRecursive": [
        {
            "before": [
                "J"
            ],
            "after": [],
            "commands": [
                "workbench.action.nextEditor"
            ]
        },
        {
            "before": [
                "K"
            ],
            "after": [],
            "commands": [
                "workbench.action.previousEditor"
            ]
        }
    ],

    //"editor.fontFamily": "'Droid Sans Mono', 'monospace', monospace",
    "editor.fontFamily": "Fira Code",
    "editor.fontLigatures": true,

Etc

• “Cursor Dark High Contrast” is a really nice high-constrast theme!

    "editor.fontWeight": "bold",

Papers

• [2303.16634] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment ¹

  • Excellent
  • Main pitch:
    • “Auto CoT evaluation”: make the LLM generate a chain-of-thoughts based on provided criteria
    • get the LLM evaluate texts based on that CoT as form-filling task
    • weight by the probabilities of the votes as given by the LLM
      • 1*chance-of-1+2*chance-of-2
  • Compares own evaluators to multiple others

• [2302.04166] GPTScore: Evaluate as You Desire²

  • basics:
    • task: e.g. summarize
    • aspect: e.g. relevance
    • eval protocol: how likely it is that this text was generated given this task and this aspect?
  • main pitch:
    • LLM-estimated likelihood of text based on task+aspect+result.
    • For summary: {Task_Specification} {Aspect_Definition} Text: {Text} Tl;dr: {Summ}
  • aspects:

• UniEval [2210.07197] Towards a Unified Multi-Dimensional Evaluator for Text Generation³

  • one trained evaluator to rule them all, based on boolean Q/A
  • “is this a coherent summary?” -> chances for yes/no -> profit

• LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models⁴

Approaches

Main goal: coherence with human judgement

Kinds

Reference-free

• LLM-estimate the probability is a generated text
  • assumption being that better texts have better probability
  • downsides: unreliable / low human correspondence according to ¹
• Ask an LLM how coherent / grammatical / criterium_name the output is from 0 to 5
  • downsides: LLMs will often pick 3 but G-Eval works around that

Reference-based

• BLEU/ROUGE and friends (n-gram based)
  • low correlation with human metrics
• (semantic) Similarity between two texts based on embeddings (BERTScore/MoverScore)
• GPTScore²: higher probability to texts better fitting the framing
• FACTUAL CONSISTENCY of generated summaries: FactCC, QAGS⁵

Framings (for GPT/LLM-based metrics)

• Form-filling (G-Eval)
• Conditional generation (=probability): GPTScore

Pairwise comparisons

• Pairwise comparisons⁴

How to evaluate evaluations

Meta-evaluators

• Meta-evaluators are a thing! (= datasets based on human ratings)
  • G-Eval¹ uses
    • SummEval ⁶
    • TopicalChat⁷
    • QAGS⁵ evaluating factual consistency/hallucinations
  • ¹
• Correlation with human metrics

Stats

• Many papers use various statistical bits to compare them.
• G-Eval (p. 6)
  • Spearman, Kendall-Tau correlations
  • Krippendorf’s Alpha

On a specific task

Task formulation

• Input:
  • Advert for a car.

    Car has 4 wheels

  • Target profile of possible client

    family with 10 kids 5 dogs living in the Australian bush

• Output:
  • Advert targeted for that profile:

    ROBUST car with 4 EXTRA LARGE WHEELS made of AUSTRALIAN METAL able to hold 12 KIDS and AT LEAST 8 DOGS

Concrete approaches

• Factual correctness / hallucinations
  • Framed as summarization evaluation: FactCC, QAGS⁵
• GPTScore, G-Eval based on yet-to-be-determined-criteria
  • Coherence, consistency, fluency, engagement, etc. — the criteria used by meta-evaluators?
  • GPTScore aspects
  • TODO: ask the car company what do they care about except factual correctness and grammar
• information density-ish? Use an LLM to get key-value pairs of the main info in the advert (number_of_wheels: 4), formulate questions based on each, and score better the adverts that contain answers to more questions!
  • Inspired by QAGS

TODO

• LLM-as-a-judge and comparative assestment
• think about human eval + meta-eval
• think about what we want from the car company
  • metrics they care about

Tried to use subfig for two figures side by side, but couldn’t \autoref it.
I had a caption for the individual subfigs but not for the large figure itself. As soon as I added the caption it worked.

  \begin{figure}%
    \centering
    \subfloat[\centering caption subfig 1]{{\includegraphics[width=0.4\linewidth]{images/fig2.png}}}%
    \qquad
    \subfloat[\centering caption subfig 2]{{\includegraphics[width=0.4\linewidth]{images/fig4.png} }}%
    \caption{Without }
    \label{fig:twosamples}%
\end{figure}

\autoref{fig:twosamples}

CVAT is a really neat labelling platform, online + free on-premise w/ Docker.
(Github: cvat-ai/cvat: Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.)

I like it more than label studio for images, has more functions, but is also “heavier” / bulkier.

Love how it supports even 600mb 4-channel TIFF satellite images and is quite fast at that.

Bits:

• Enable auto-save every N minutes
• <C-a> for snipping polygons to existing polygon ponits
• “Backup project” to re-import later, “Export project” to get e.g. YOLO annotations

Pydantic’s FilePath is like Path except that the file has to exist and be a file.

BUT FilePath when validating expects a string as input, not a Path! (in other words: FilePath(Path) doesn’t seem to work)

So when I create a Validator that converts str into Path ¹:

@field_validator("filename", mode="before") 
@classmethod 
def parse_filename(cls, value: str | Path) -> Path: 
	return Path(value)

I get a wonderful

>       doc = UCFDocument.model_validate_json(json_string)
E       pydantic_core._pydantic_core.ValidationError: 1 validation error for UCFDocument
E       filename
E         Input is not a valid path for <class 'pathlib.Path'> [type=path_type, input_value=PosixPath('/home/sh/w/cor...n/doc.pdf_data/doc.pdf'), input_type=PosixPath]

tests/ucf/test_data_structures.py:179: ValidationError

Again, the error is a PosixPath not being a Path, though it is one:

E         Input is not a valid path for <class 'pathlib.Path'> [type=path_type, input_value=PosixPath('/home/sh/w/cor...n/doc.pdf_data/doc.pdf'), input_type=PosixPath]

# explicitly expecting a PosixPath creates an even better
E         Input is not a valid path for <class 'pathlib.PosixPath'> [type=path_type, input_value=PosixPath('/home/sh/w/cor...n/doc.pdf_data/doc.pdf'), input_type=PosixPath]

Not intuitive at all.

Solution is to give FilePath strings and only strings, or drop FilePath to begin with.

├── pydantic v2.10.6
│   ├── annotated-types v0.7.0
│   ├── pydantic-core v2.27.2
│   │   └── typing-extensions v4.12.2

The Open LLM Leaderboard is dead¹, as good time as any to look for new eval stuff!

• HF universe

  • The Open LLM Leaderboard people are actually OpenEvals (OpenEvals), and they created other cool stuffs
    • huggingface/lighteval: Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends eval suite
      • I like their documentation for new task / new model
    • huggingface/evaluation-guidebook: Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
      • the contents are awesome, w/ examples, model-as-a-judge, etc.
  • evaluate-metric (Evaluate Metric) recommended by them as a guide to existing metrics

• Harnesses

  • Evalverse: Unified and Accessible Library for Large Language Model Evaluation meta-thing that can run different harnesses based on the target. Github repo archived? UpstageAI/evalverse: The Universe of Evaluation. All about the evaluation for LLMs.
  • open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
    • don’t really like their documentation on adding datasets/models
  • The venerable stanford-crfm/helm: Holistic Evaluation of Language Models (HELM)
  • openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

• Resources / articles:

  • How to evaluate by Meta: Validation | How-to guides
  • HF eval guidebook: huggingface/evaluation-guidebook

• M.2 slots have keys: How do I install an M.2 SSD on my computer? - Transcend Information, Inc.
• B+M means both B and M slots are acceptable for the B+M module

CLI: sudo lshw -C disk tells you all disks h

In fsspec fs.copy() doesn’t really work from local to remote, also existing or not-existing directories etc.

Their documentation has a whole page on this: Copying files and directories — fsspec 2024.10.0.post13+gdbed2ec.d20241115 documentation

GitLab CLI - glab | GitLab Docs: CLI thingy to interact with gitlab.

It’s really neat and has a cool CLI interface, either you set things through flags or you get a neat menu to choose from!

Smart enough to parse current directory!

Pipelines:

• glab ci status for a curses interface
• glab ci status -l is a live view as they run

Git Tag: A Tutorial for Tagging Releases in Git - DEV Community

#ligthweight tag
git tag v1.0.0

# full
git tag -a v1.0.0 -m "Releasing version v1.0.0"

Tags don’t get pushed automatically. For this, git push origin v1.0.0

• Writing your pyproject.toml - Python Packaging User Guide
• uv uses hatch which has this to say about dynamic versioning: Configuring project metadata - Hatch

[project] 
dynamic = ["version"] 

[tool.hatch.version] 
path = "..."

Path is a python file w/ version info. If using src layout, src has to be included in the path.j

For uv, this works. Described for example here: Versioning Python Projects with Hatch (I like __init__.py though, not about as that guide does)

uv-dynamic-versioning · PyPI exists but I don’t really see why.

Because every single goddamn time

Command line: How do you rotate a PDF file 90 degrees? - Unix & Linux Stack Exchange

pdftk input.pdf cat 1-endwest output output.pdf

1- is needed because page range, here for all pages.

endwest etc from man page:

 [<begin page number>[-<end page number>[<qualifier>]]][<page rotation>]

The qualifier can be even or odd, and the page rotation can be north, south, east, west, left, right, or down.
Each option sets the page rotation as follows (in degrees): north: 0, east: 90, south: 180, west: 270, left: -90, right: +90, down: +180. left, right, and down make relative adjustments to a page’s rotation.

• Just got sold on this as a way to automate creation of config maps, secrets and friends!
  • Though allegedly hard-ish to learn
• Need to look deeper.

This is brilliant: Git, rewrite previous commit usernames and emails - Stack Overflow

TL;DR

git config --global alias.change-commits '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f"

Then

git change-commits GIT_AUTHOR_NAME "old name" "new name"

# last 10 commits
git change-commits GIT_AUTHOR_EMAIL "old@email.com" "new@email.com" HEAD~10..HEAD

Depending on why I need this, I may need also GIT_COMMITTER_[NAME/EMAIL]

For multiple times, I created an change-commit-f that forces overwiriting the backup:

git config --global alias.change-commits-f '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch -f --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f"

A quick tldr from other answer, may be better but untested:

git config alias.change-commits '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f "
git change-commits GIT_AUTHOR_NAME "<Old Name>" "<New Name>" -f
git change-commits GIT_AUTHOR_EMAIL <old@email.com> <new@email.com> -f
git change-commits GIT_COMMITTER_NAME "<Old Name>" "<New Name>" -f
git change-commits GIT_COMMITTER_EMAIL <old@email.com> <new@email.com> -f

(Previously: 220408-1822 Gitlab ‘you cannot push commits for ..’ error)

Lines of code that beat A/B testing (2012) | Hacker News / 20 lines of code that will beat A/B testing every time

TL;DR: A/B testing that automatically increases how often “good” versions are shown

First impressions

• INCEpTION User Guide

  • Docker works:
    • docker run -it --name inception -p8080:8080 ghcr.io/inception-project/inception:35.1
    • $ docker run -it --name inception -v /srv/inception:/export -p8080:8080 ghcr.io/inception-project/inception:35.1
      • data will be in /srv/inception

• Creating a project automatically fills it with sample data:

• Tagsets

  • Export format is:

  {
    "name" : "BBK",
    "description" : null,
    "language" : null,
    "tags" : [ {
      "tag_name" : "aaa_human_processed",
      "tag_description" : null
    }, {
      "tag_name" : "block",
      "tag_description" : null
    } ],
    "create_tag" : false
  }
  

  • Import format: I can get it do do only txt, one tag per line, first line is name of tagset

• A layer has to be linked to a feature (string) which then can be linked to a tagset: (INCEpTION User Guide)

  • then you can add keybindings manually
  • and the “editor type” for the tag list is neat, “Radio group” works nicely for tagsets it doesn’t consider small

• annotations get saved automatically

• in the viewer, you can set dynamic for annotations differing based on color

Import/Export

• Admin->Export can
  • export the entire projects
  • project + separately a copy of the anntations
• To import a project: admin->projects, where you can create a new project, you can also import. You import the .zip without extraction

Features

• Supports overlapping annos
• Supports annos across page boundaries
• Doesn’t support gaps in annos!

Resources

• Not-official but really nice tutorial: 12. INCEpTION — New Languages for NLP

Using uv as your shebang line – Rob Allen (HN comments) and more detailed article on this: Lazy self-installing Python scripts with uv

But especially Defining Python dependencies at the top of the file – Rob Allen and the PEP 723 – Inline script metadata | peps.python.org

You can add uv to the shebang line as

#!/usr/bin/env -S uv run --script

And you can set requirements by adding this under the shebang line:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "flickrapi",
# ]
# ///

Then you can uv run sync-flickr-dates.py

Full package:

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "flickrapi",
# ]
# ///
import flickrapi
print("\nI am running")

❯ chmod +x test.py
❯ ./test.py
Installed 11 packages in 134ms

I am running!

Neat!

git - How to cherry-pick multiple commits - Stack Overflow:

For one commit you just pase its hash.
For multiple you list them, in any order.
For a range, you do oldest-latest but add ~, ^ or ~1 to the oldest to include it. Quoting directly from the SO answer:

# A. INCLUDING the beginning_commit
git cherry-pick beginning_commit~..ending_commit
# OR (same as above)
git cherry-pick beginning_commit~1..ending_commit
# OR (same as above)
git cherry-pick beginning_commit^..ending_commit 

# B. NOT including the beginning_commit
git cherry-pick beginning_commit..ending_commit

So, given that kubectl cp was never reliable ever for me, leading to many notes here, incl. 250115-1052 Rancher much better way to copy data to PVCs with various hacks and issues like 250117-1127 Splitting files, 250117-1104 Unzip in alpine is broken issues etc. etc. etc.

For many/large files, I’d have used rsync, for which ssh access is theoretically needed. Not quite!

rsync files to a kubernetes pod - Server Fault

ksync.sh (EDIT Updated by ChatGPT to support files with spaces):

if [ -z "$KRSYNC_STARTED" ]; then
    export KRSYNC_STARTED=true
    exec rsync --blocking-io --rsh "$0" "$@"
fi

# Running as --rsh
namespace=''
pod=$1
shift

# If user uses pod@namespace, rsync passes args as: {us} -l pod namespace ...
if [ "X$pod" = "X-l" ]; then
    pod=$1
    shift
    namespace="-n $1"
    shift
fi

# Execute kubectl with proper quoting
exec kubectl $namespace exec -i "$pod" -- "$@"

Usage is same as rsync basically :

./ksync.sh -av --info=progress2 --stats /local/dir/to/copy/  PODNAME@NAMESPACE:/target/dir/

(Or just --progress for per-file instead of total progress).

Rsync needs to be installed on server for this to work.

For flaky connections (TODO document better): -hvvrPt --timeout1 and while ! rsync ..; do sleep 5; done ¹

TL;DR pipx inject target_app package_to_inject

• pipx is awesome, and can install apps in their own virtualenv
• If you try e.g. pipx psutil it refuses, it’s a library, not an app
  • and tells you to use pip
• If I want psutil for the MemoryGraph widget in (pipx install-ed) qtile, that doesn’t help
• What does: pipx inject qtile psutil

❯ pipx inject qtile psutil
  injected package psutil into venv qtile
done! ✨ 🌟 ✨

If no real config thingy is required/wanted, then this works (stolen from Parsing Dictionary-Like Key-Value Pairs Using Argparse in Python | Sumit’s Space)¹:

def parse_args():
    class ParseKwargs(argparse.Action):
        def __call__(self, parser, namespace, values, option_string=None):
            setattr(namespace, self.dest, dict())
            for value in values:
                key, value = value.split("=")
                getattr(namespace, self.dest)[key] = value
    parser.add_argument("--no-pics", action="store_true", help="Predict only on videos")
	# ...

    parser.add_argument(
        "-k",
        "--kwargs",
        nargs="*",
        action=ParseKwargs,
        help="Additional inference params, e.g.: batch=128, conf=0.2.",
    )

#!/bin/bash

BATTINFO=$(acpi -b)
LIM="00:15:00"
if  grep Discharging) && $(echo $BATTINFO | cut -f 5 -d " ") < $LIM ; then
  # DISPLAY=:0.0 /usr/bin/notify-send "low battery" "$BATTINFO"
  dunstify "low battery" "$BATTINFO"
fi

For this, install and run on startup dunst, then cron job for the above.

• Assigning Pods to Nodes | Kubernetes
  • Operators

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/hostname
                operator: NotIn
                values:
                  - node_to_avoid

(operator: In for the list of the allowed nodes)

kubectl get pods -o wide shows the nodes

E.g. to upload it somewhere where it’s hard to upload large files

See also: 250117-1104 Unzip in alpine is broken

# split
split -b 2G myfile.zip part_
# back
cat part_* > myfile.zip

docker - Why do I get “unzip: short read” when I try to build an image from Dockerfile? - Stack Overflow:

TL;DR alpine’s unzip is busyboxes, and fails for me with

/data/inference_data # unzip rd1.zip
Archive:  rd1.zip
unzip: short read

apk add unzip installs the same real one I have on all other computers, and then it works.

Related: 250115-1238 Adding wandb to a CLI yolo run

• Reference are surprisingly hard to find on the website: results - Ultralytics YOLO Docs

• yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=500 imgsz=640 device=0,1 name=yolo11s-aug-500epochs-full

• YOLOv11 sets default batch_size 16, one can set -1 for it to automatically pick one that’s 60% of GPU, or 0.8 to automatically pick one that’s 80% of GPU

• To decrease verbosity in predictions, verbose=False to model.predict() (and `.track()) works¹.

• Changing imgsz= to something lower may not necessarily make it faster, if a model was trained with a certain size it may predict faster at that size (e.g. OSCF/TrapperAI-v02.2024 predicts at 40+ iterations per second when resized to 640 and ~31 when left to its default 1024pd)

  • Resizing (if provided a single int, not a tuple) works by making the larger side of the image equal to the given one, if padding is needed grey is used(?)

• Half-life precision (if supported by GPU) is really cool! half=True makes stuff faster (no idea about prediction quality yet)

  • And batch size obviously

• vid_stride predicts every Nth video frame, was almost going to write that myself

All-in-all I like ultralytics/YOLO

Assuming you’re doing a YOLO run w/

yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=500 imgsz=640 device=0,1 name=yolo11s-aug-500epochs-full

• name there becomes training run name in wandb + directory name in /data/projct
  • (project on wandb will be `-data-project-)
• To enable wandb:

pip install wandb
yolo settings wandb=True
wandb login 

Or if you’re inside an ultralytics:latest Docker container,

apt install -y bash screen
bash
pip install wandb
yolo settings wandb=True
wandb login 
screen 
yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=500 imgsz=640 device=0,1 name=yolo11s-aug-500epochs-full

Also useful:

# get a model file
wandb artifact get /proje:ject/run_alxxxpy7_model:v0 --root target_director/

TL;DR:
git clean -d -f .

If a broken clone / switch leaves stray broken files: error: The following untracked working tree files would be overwritten by checkout:, that fixes it.¹

I had exotic not enough shared memory crashes, ty GC for giving me these lines I do not yet understand but that seem to work, later I’ll dig into why (TODO)

apiVersion: v1
kind: Pod
metadata:
  name: CHANGEME
  namespace: CHANGEME-ns
spec:
  restartPolicy: Never
  containers:
    - name: sh-temp-yolo-container-3
      image: ultralytics/ultralytics:latest
      command: ["/bin/sh", "-c"]
      args: 
        - "yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=30 imgsz=640 device=0,1"
      resources:
        requests:
          nvidia.com/gpu: "2" # GPUs for each training run
          ephemeral-storage: "12Gi"
        limits:
          nvidia.com/gpu: "2" # same as requests nvidia.com/gpu
          ephemeral-storage: "14Gi"
      volumeMounts: # Mount the persistent volume
        - name: data
          mountPath: /data
        - name: shared-memory
          mountPath: /dev/shm
  volumes: 
    - name: shared-memory
      emptyDir:
        medium: Memory
    - name: data
      persistentVolumeClaim:
        claimName: sh-temp-yolo-pvc

Both requests AND limits, as well as mount shared memory in volumeMounts + volumes.

For one-off HTML exports, found the plugin KosmosisDire/obsidian-webpage-export: Export html from single files, canvas pages, or whole vaults. Direct access to the exported HTML files allows you to publish your digital garden anywhere. Focuses on flexibility, features, and style parity.

It exports both the vault and individual pages, and adds things like toc on the left and toggles and optionally file browsing. Much better than the other pandoc-based export plugin that I could not get to work reliably for exporting good-looking HTML

• Qutebrowser:
  :set colors.webpage.darkmode.enabled true Really neat actually! ALSO: colors.webpage.preferred_color_scheme: dark tells websites my preference
• Nvim: :colorscheme zaibatsu
• Redshift: redshift -r -P -O 4000 -b 0.3
• If I’m doing all that, I probably want to mute my speakers as well

Nested list comprehensions are a horrible idea because they are hard to parse, and I never understood them, BUT.¹

python - How do I make a flat list out of a list of lists? - Stack Overflow has a discussion in the accepted answer about the suggested syntax to flatten lists, and I get it now.

flat_list = [
    x
    for xs in xss
    for x in xs
]

# equivalent to 
flat_list = []

for xs in xss:
    for x in xs:
        flat_list.append(x)

So,

[x for xs in xss for x in xs]

Comments:

I found the syntax hard to understand until I realized you can think of it exactly like nested for loops. for sublist in l: for item in sublist: yield item

[leaf for tree in forest for leaf in tree]

I kept looking here every time I wanted to flatten a list, but this gif is what drove it home: i.sstatic.net/0GoV5.gif 

GIF IN QUESTION, after which it clicked for me:

The first element is the one that gets returned!

for tree in forest: for leaf in tree: return leaf
[leaf (for tree in forest, for leaf in tree)]
[leaf (for tree in forest for leaf in tree)]
[leaf for tree in forest for leaf in tree]

Found Understanding nested list comprehension syntax in Python — /var/ which expands on this, quoting PEP

It is proposed to allow conditional construction of list literals using for and if clauses. They would nest in the same way for loops and if statements nest now.

It then shows:

for x in non_flat:
    if len(x) > 2
        for y in x:
            y
# equivaent to
>>> [ y for x in non_flat if len(x) > 2 for y in x ]

MIND. BLOWN.

I’m not sure “this requires you to understand Python syntax” is an argument against using a given technique in Python This is about itertools.chain(*list, which is the way to go imo. But still, * is python syntax, otherwise there are more or less readable ways to do thigs and nested comprehensions are rarely worth it

TL;DR clone to unencrypted directory

error: unable to create file datasets/processed/GitHub-Mensch-Animal_Finetuned/data/val/labels/1713256557366,hintergrund-meister-lampe-geht-das-licht-aus-vom-rueckgang-der-arten-tierische-und-pflanzliche-neubuerger-108~v-16x9@2dM-ad6791ade5eb8b5c935dd377130b903c4b5781d8.txt: File name too long

error: cannot stat ‘datasets/processed/GitHub-Mensch-Animal_Finetuned/data/val/images/1713256557366,hintergrund-meister-lampe-geht-das-licht-aus-vom-rueckgang-der-arten-tierische-und-pflanzliche-neubuerger-108~v-16x9@2dM-ad6791ade5eb8b5c935dd377130b903c4b5781d8.jpg’: File name too long

The usual solution¹ is to set longpaths = true in git config or during clone (git clone -c core.longpaths=true <repo-url>)

Didn’t solve this for me.

BUT apparently my encrypted $HOME has something to do with this, because filenames can get longer (?) in this case and go over the limit?.. git checkout-index: unable to create file (File name too long) - Stack Overflow

And one solution is to clone to /tmp or whatever is not encrypted by encryptfs.

(And in my case I could rename these files in a commit in /tmp and after that it worked, as long as I don’t check out the revisions with the long filenames)

• Goal: centerline trace pngs
• Inscape
  • Script, extension, now part of Inkscape: inkscape-centerline-trace/centerline-trace.py at master · fablabnbg/inkscape-centerline-trace
  • Inkscape as a purely command line tool : r/Inkscape reddit on using the CLI and running the extension standalone and how it doesn’t really work
  • Can’t get it to work using CLI either, there’s just bitmap-trace that has no centerline option
• Not inkscape
  • autotrace/autotrace: bitmap to vector graphics converter
  • Potrace
    • no centerline

Autotrace is awesome!

This alone works really nicely:

autotrace -centerline AMPERSAND.png  -output-file AMPERSAND.svg

Fish script for batch processing, courtesy of ChatGPT:

#!/usr/bin/fish

# Check if autotrace is installed
if not type -q autotrace
    echo "autotrace is not installed. Please install it first."
    exit 1
end

# Loop through each .png file provided as an argument
for file in $argv
    # Check if the file extension is .png
    if string match -r '\.png$' $file
        # Set the output filename by replacing .png with .svg
        set output_file (string replace -r '\.png$' '.svg' $file)

        # Execute autotrace with centerline option
        autotrace -centerline $file -output-file $output_file

        # Confirmation message
        echo "Processed $file to $output_file"
    else
        echo "Skipping $file: not a .png file"
    end
end

And a more simple one:

#!/usr/bin/fish
for file in $argv
    autotrace -centerline $file -output-file "$file.svg"
end

Options

ChatGPT says this:

autotrace -centerline -input-format png -output-format svg -output-file traced_dejavu.svg -dpi 300 -error-threshold 0.5 -corner-threshold 85 -filter-iterations 2 -noise-removal 0.99 -line-threshold 0.5 -corner-surround 3

(et 1 is best)

Using the Command Line - Inkscape Wiki

• inkscape action-list shows all available actions

• man inkscape is the latest and best

• inkscape AMPERSAND.png --export-type="svg" --export-area-page --batch-process works but asks me about import options

inkscape --shell, man page gives examples:

file-open:file1.svg; export-type:pdf; export-do; export-type:png; export-do
                   file-open:file2.svg; export-id:rect2; export-id-only; export-filename:rect_only.svg; export-do

OK this works for no questions about how to import it:

> file-open:AMPERSAND.png
> export-filename:AM.svg
> export-do

• Quarto markdown is a superset of pandoc markdown, unrelated article about the latter: Everything pandoc Markdown can do
• Pandoc has a python library: Pandoc (Python Library)
• panflute is a more python way to parse docs and write pandoc filters, and I love it: User guide — panflute 2.3.0 documentation

I missed an ability to recursively look for elements matching a condition in panflute, so:

def _recursively_find_elements(
    element: Element | list[Element], condition: Callable
) -> list[Element]:
    """Return panflute element(s) and their descendants that match conditition.
    """
    results = list()

    def action(el, doc):
        if condition(el):
            results.append(el)

    if not isinstance(element, list):
        element = [element]

    for e in element:
        e.walk(action)


    return results

# sample condition
def is_header(e) -> bool:
	cond = e.tag == "Header" and e.level == 2  # and "data-pos" in e.attributes
	return cond

Ah, to read:

ddoc = pf.convert_text(
	markdown,
	input_format="commonmark_x+raw_html+bracketed_spans+fenced_divs+sourcepos",
	output_format="panflute",
)

To output readably:

pf.stringify(el).strip() 

Pandoc/panflute get line numbers of elements

• input_format has to be commonmark[_x]+sourcepos
  • sourcepos isn’t too well documented, only w/ commonmark
  • it basically sets el.attributes['data-pos'] a la 126:1-127:1
  • line_no always matching what I expect

def _parse_data_pos(p: str) -> tuple[tuple[int, int], tuple[int, int]]:
	"""Parse data-pos string to (line, char) for start and end.
	
	Example: '126:1-127:1' -> ((126, 1), (127, 1))
	
	Arguments:
		p: data-pos string as generated by commonmark+sourcepos extension.
	"""
	start, end = p.split("-")
	start_l, start_c = start.split(":")
	end_l, end_ch = end.split(":")
	return (int(start_l), int(start_c)), (int(end_l), int(end_ch))

One control, play/pause.

const buttons = document.querySelectorAll('.play-pause-btn');

buttons.forEach(button => {
    const audio = document.getElementById(button.dataset.audio);

    button.addEventListener('click', () => {
        if (audio.paused) {
            // Pause all other audio files
            document.querySelectorAll('audio').forEach(a => {
                a.pause();
                a.currentTime = 0; // Reset other audio files
            });
            document.querySelectorAll('.play-pause-btn').forEach(btn => {
                btn.textContent = '▶';

            });

            // Play the clicked audio
            audio.play();
            button.textContent = '⏸︎';

        } else {
            audio.pause();
            button.textContent = '▶';
        }
    });

    // Reset button icon when audio ends
    audio.addEventListener('ended', () => {
        button.textContent = '▶';
    });
});

Multiple players:

<div class="player-container">
	<button class="play-pause-btn" data-audio="audio1">▶️</button>
	<audio id="audio1" src="audio1.mp3"></audio>
</div>

<div class="player-container">
	<button class="play-pause-btn" data-audio="audio2">▶️</button>
	<audio id="audio2" src="audio2.mp3"></audio>
</div>

.player-container {
    display: inline;
    vertical-align: text-bottom;
    align-items: center;
    margin-bottom: 20px;
}
.play-pause-btn {
    font-size: 32px;
    background: none;
    border: none;
    cursor: pointer;
    margin-right: 10px;
}

Fish Shell function for sourcing standard .env files :

 . (sed 's/^/export /' .env | psub)

(And yet another mention of Taskfile that I’ll definitely look into nowG)

I want to automatically get the PDF version of quarto/reveal presentations. The usual way would be to open the presentation in export mode e, then print with no margins through the usual print window.

I want to do this automatically as part of a CI/CD pipeliene.

Options

• Generate PDF from revealjs presentation (consistent with PDF Export in revealjs) · quarto-dev/quarto-cli · Discussion #7018
• selenium-print · PyPI / bubblegumsoldier/selenium-print uses selenium+chromium to do this.
• PDF Export | reveal.js
  • Support other PDF export tools for revealjs (e.g decktape) · Issue #4677 · quarto-dev/quarto-cli

Selenium-print

selenium-print · PyPI / bubblegumsoldier/selenium-print uses selenium+chromium to do this.

As for the printing options available in Chrome, this looks relevant:

selenium-print/seleniumprint/drivers/chrome_pdf_driver.py at main · bubblegumsoldier/selenium-print

        pdf = self.driver.execute_cdp_cmd("Page.printToPDF", {"printBackground": True})

OK, so it’s all a static option.

Chrome DevTools Protocol - Page domain has the other available options — which is what I need.

The rest of the code feels like a wrapper to this — maybe I can drop the entire library and just use these single bits?

Decttape (?)

• Generate PDF from revealjs presentation (consistent with PDF Export in revealjs) · quarto-dev/quarto-cli · Discussion #7018

• Command substitutions and variables: Fish for bash users — fish-shell 3.7.1 documentation
• string collect: string-collect - join strings into one — fish-shell 3.7.1 documentation
• SO: Read file into environment variable? · Issue #7323 · fish-shell/fish-shell:

set VARNAME (cat ~/myfile | string collect)

Here string collect makes sure it’s a multiline variable instead of an array composed of one element per line.

TL;DR use your own personal settings, then “dev settings” (!), then create one but set the resource owner to the organization.

(As of 2024-10-14. Hard to find clear AND CORRECT documentation on this.)

Create access token for organization · community · Discussion #74701.

Gitlab mirroring didn’t work for me after trying for hours, I give up. CI/CD it is.

• Refs:
  • Predefined CI/CD variables reference | GitLab
  • How do I push to a repo from within a gitlab CI pipeline? - Stack Overflow

• Font Finder By Image — Fontspring Matcherator allows manually specifying glyphs and shows all paid fontns
• WhatTheFont Font Finder - Identify Fonts by Image is the very best, instantly parsed the noisy image and the fonts looked much closer (and found some very famous ones missed by the first)

On a slow connection or other constraints, you can check out only the last N commits or only a specific branch:

git clone --depth 3 --branch some-branch https://some-repo.org

Git - git-clone Documentation

TIL¹:

• ./git/info/exclude is your local .gitignore outside the repository tree!
• git update-index --assume-unchanged .gitignore makes git stop checking the changes for that file in the working tree. --no-assume-unchanged to take it back.²³

Finally a place for my local ignores that won’t appear in autocomplete suggestions for git add and friends. In Pycharm I have changelists, and now I finally have a solution for my just-as-usual vim/git/CLI workflow as well.

BUT:

• exclude won’t work if the file is already tracked (says SO but for me it works?..)

As of 2024-10-02 20:16 at least.

give me the ag command to look inside markdown and yaml files only

GPT4o¹:

`ag --include='*.md' --include='*.yaml' --include='*.yml' 'search_pattern'`

GPT4²:

`ag "search_pattern" --markdown --yaml`

github/git-sizer: Compute various size metrics for a Git repository, flagging those that might cause problems (linked by Repository limitations and recommendations)

TL;DR too many files? too large one? too many tags? Find out!

yay took forever to compress pycharm. Like, 5-10 mins at least.

TODO documentation, but if you don’t care about package size but care about speed, this will speed up everything considerably:

PKGEXT='.pkg.tar' yay -S pycharm-professional

Wanted to do sth like this all the time, but the help basically told me to copypaste multiple arguments etc.

Will come back to bite me but nice to have the option I guess

Question: How can I install packages without having to confirm? · Issue #1033 · Jguer/yay:

echo y | LANG=C yay --noprovides --answerdiff None --answerclean None --mflags "--noconfirm" $PKGNAME

(--noconfirm is not documented in help and man, and of course can create problems¹)

BUT ALSO

yes is a command that exists.

Then: yes | yay ... or yes | LANG=C yay

And generally really neat to have a command for answering “yes” to prompts.

And then quick unsafe bad dangerous command to update everything that includes 241001-1512 Yay better compression algos:

echo y | LANG=C PKGEXT='.pkg.tar'  yay  --answerdiff None --answerclean None --mflags "--noconfirm"

• You can share lists in Google Maps, making them editable by everyone with a google account
• Google My Maps is much cooler, with layers etc

Google My Maps

• Really nice tutorial on everything: Visualize your data on a custom map using Google My Maps – Google Earth Outreach
• To share it editable, you share -> Share on Drive
• drag and drop doesn’t always work
• style and colors
  • layer can be styled uniformly (easy)
  • style by data column: per-field, e.g. description, like a groupby
  • individual styles per element
• you can drag and drop items between layers only if they’re styled individually??
• If you style by e.g. description, assign icons/colors, then style individually to move them, then back to style by description, it seems to save the previous settings

In pycharm, updated black to use these args: --preview --enable-unstable-feature string_processing $FilePath$

The (future of the) Black code style - Black 24.8.0 documentation has both the preview style and the unstable features one can enable (by passing the flag multilple times).

string_processing breaks long strings into multiple shorter ones, one on each line.

• Useful:

  • Running mypy and managing imports - mypy 1.12.0+dev.ecfab6a02415c46eda5717ec6ee9bfac8115c1e9.dirty documentation evergreen
  • Error codes - mypy 1.12.0+dev.ecfab6a02415c46eda5717ec6ee9bfac8115c1e9.dirty documentation
  • The mypy configuration file - mypy 1.12.0+dev.ecfab6a02415c46eda5717ec6ee9bfac8115c1e9.dirty documentation

• mypy cares about packages, especially __init__.py files — the uppermost dir with such a file will be the root package. See Mapping files to modules - Running mypy and managing imports - mypy 1.12.0+dev.ecfab6a02415c46eda5717ec6ee9bfac8115c1e9.dirty documentation

• This is needed to do per-package configs

[mypy-mycode.package.etc]
disable_error_code = attr-defined, union-attr

• Misc
  • tuple[Path, ...] means tuple containing any number of Path objects

• Problem: the pycharm extension I have crashes in the last version. :(

• Main and best: python - How to run Pylint with PyCharm - Stack Overflow

  • idea of multiple ’external tools’ for diff pylint options: HOWTO - Setup pylint with PyCharm and the PyLint Plugin

My changes:

• arguments: --msg-template="{abspath}:{line:5d},{column:2d}: {C}/{msg} ({symbol})" --output-format=colorized "$FilePath$"
  • note ", otherwise it failed for me, and {C} — for the message class see man page (or below) for list of format string options.
• output filters for on-click: $FILE_PATH$:\s*$LINE$\,\s*$COLUMN$

I had two options as separate tools:

• pylint for a single file
• pylint for the module — same as above, but
  • with $FileParentDir$ at the end.
  • --recursive=y — fails on no __init__.py otherwise

Worked neatly with a .pylintrc file in repo root with e.g.

[tool.pylint.main]

[tool.pylint.basic]
# "too-few-public-methods", disable
min-public-methods=0

function-naming-style="camelCase"
argument-naming-style="camelCase"
method-naming-style="camelCase"
variable-naming-style="camelCase"
attr-naming-style="camelCase"

[tool.pylint."messages control"]
disable = [
    "fixme",  # TODOs
    "import-error",  # runner has them in its environment
    "import-outside-toplevel",  # explicit requirement of XXX to import where used
    "duplicate-code"  # entangling different extensions/modules is not the solution
]

Pylint format string options from man pylint:

path   relative path to the file

abspath
	  absolute path to the file

line   line number

column column number

end_line
	  line number of the end of the node

end_column
	  column number of the end of the node

module module name

obj    object within the module (if any)

msg    text of the message

msg_id the message code (eg. I0011)

symbol symbolic name of the message (eg. locally-disabled)

C      one letter indication of the message category

category
	  fullname of the message category

For example, the former (pre 1.0) default format can be obtained with:

  pylint --msg-template='{msg_id}:{line:3d},{column}: {obj}: {msg}'

Bonus

If I have this post open, I’ll need this one anyway: Messages control - Pylint 4.0.0-dev0 documentation

So Slicer seems to use both and I need to as well, so I’ll have to learn that sooner or later.

• VTK
  • VTK advertises their Book, available online, which looks like an excellent intro about everything, terminology etc.
    • Chapter 5 - Data Representation — VTK Book
  • wiki: VTK/Tutorials - KitwarePublic but superficially less useful to me?..
• ITK

  • open-source, cross-platform library that provides developers with an extensive suite of software tools for image analysis (About | ITK)

  • Python Quick Start - ITK’s documentation

  • Examples — v5.4.0

    • Common especially has interesting things like add noise etc.: Examples — v5.4.0

  • there’s a python package docu:

    • Quick start guide — ITKPythonPackage documentation
• Slicer
  • Slicer examples as usual as always: Script repository — 3D Slicer documentation
    • Segment effects! Script repository — 3D Slicer documentation
      • Fills all internal holes inside segments. For example, fills in empty regions inside vertebral bodies segmented with Grow from seeds with intensity-based masking.
        • shows many other cool useful things as well, incl. adding segments

Is ‘file’ a keyword in python? - Stack Overflow

TL;DR python3 doesn’t care about file, regardless of what syntax highlighters think about this.

TL;DR:

• pydocstyle != flake8-docstrings · PyPI!
• read the pypy documentation
• docstring-convention=google in flake8 config file, ignores there as well together with the rest.

pydocstyle / python-flake8-docstrings is a thing. Forgot I had it installed and spent a lot of time trying to understand pycharm’s output

Usage — pydocstyle 0.0.0.dev0 documentation flake8-docstrings · PyPI

To ignore things, you don’t do:

[pydocstyle]
convention = google
ignore = D401

It’s either ignore or convention. Which quietly happened in the background, and I thought it doesn’t read my config file since D401 was still shown.

Correct would be this:

[pydocstyle]
convention = google
add-ignore = D401

EDIT GodDAMN it, pydocsyle parsing a config file doesn’t mean that flake8(-..docstring) will.

Reading the flake8 plugin docs, I should add THIS and to flake8 config file. Ignores also are set up there using the usual way.

docstring-convention=google

And the pydoctest config file search and add-ignore is irrelevant. God lost so much time to this

EDIT: this may be all wrong, in the debugger I can just edit the code and it gets automatically used by slicer — logical, since it’s just a CLI script.

It’s the .xml with the interface that’s problematic and it can’t be reloaded using the method below, or at least I couldn’t.

A scriptedcli module imported to Slicer doesn’t show for me the usual “reload” buttons as seen in scripted modules in dev. mode. To develop, I need my changes I need to reload it w/o restarting 3dslicer.

Based on this more complete example¹ linked in the script repository²

>>> mpath = "/full/path/to/the/my_module.py"
>>> factoryManager = slicer.app.moduleManager().factoryManager()
>>> factoryManager.registerModule(qt.QFileInfo(mpath))
>>> factoryManager.loadModules(["my_module"])
True

BONUS:

• To debug such a module using pycharm, adding the pydev lines inside it works (and official slicer debug extension doesn’t):

import pydevd_pycharm
pydevd_pycharm.settrace('localhost', port=5678, stdoutToServer=True, stderrToServer=True)

Links

• GUI
  • XML reference Documentation/Nightly/Developers/SlicerExecutionModel - Slicer Wiki

Debugging

TL;DR Pycharm Professional instructions work

• These instructions: SlicerRt/SlicerDebuggingTools: Extension for 3D Slicer containing various tools for module development and debugging¹
  • No need to change anything in module code
• During debugging everything freezes
• You can use pycharm python CLI to look around etc. — neat!

not TL;DR

• SlicerRt/SlicerDebuggingTools: Extension for 3D Slicer containing various tools for module development and debugging¹
  • needs pycharm Pro
  • SlicerDebuggingTools/PyDevRemoteDebug/PyDevRemoteDebug.py at master · SlicerRt/SlicerDebuggingTools
  • source of the 3dslicer bits
• pdb-attach · PyPI can attach to random programs,
  • os.getpid() is my friend
• Documentation/4.5/Developers/FAQ/Python Scripting - Slicer Wiki at some point it was possible to use python-remote-pdb
• ionelmc/python-remote-pdb: Remote vanilla PDB (over TCP sockets). 2y ago last commit
• I can use these same instructions mostly for pdb-attach, I see no reason it shouldn’t work, python is python
• it Kiiiinda works but breaks after the first button
• Giving up for now
• Update: installed Pycharm Pro, works!
  • Pycharm Pro egg was in /opt/pycharm-professional/debug-eggs/pydevd-pycharm.egg

Basics

Starting Slicer

Loads the module automatically (for some reason doing it manually doesn’t preserve it across restarts?..)

Slicer  --additional-module-path /path/to/my/extension/one/lower/than/what/i/would/pick/in/Slicer/GUI

Logging

• Not directly mentioned in the help because they state it’s always better to use the debugger²
• Slicer/Base/Python/slicer/slicerqt.py at main · Slicer/Slicer has the slicer setup
• I can do logging.warning("test") in the python console and it works
• Slicer settings allow changing the loglevel of its python console!

Module types: CLI modules

Different types of modules exist: Module Overview — 3D Slicer documentation

In the first tests there was a lot of latency.

Solution: CLI modules (not extensions) that can run in the background and whose status can be queried, which can be added to new extension from extension editor.

Now I understad this better:

• CLI modules are simple I/O, non-blocking, that’s where the networkig logic should go to
• scripted are python scripts with GUIs

• This bit about running CLI in the backgroundPython FAQ — 3D Slicer documentation

Looking into the sample code

Module of type scripted

• Created an extension of type cmake, to which I added a module of type scripted. It’s the sample thresholding extension
  • Base class: Slicer/Base/Python/slicer/ScriptedLoadableModule.py at main · Slicer/Slicer
• Sample data
  • It shows how to add sample data to the extension for testing
  • It can then be found in the Sample Data module, named after the extension itself!

Module of type scriptedCLI

• I hope it’s the same as type CLI, but only with python

• I think it’s lassoan/SlicerPythonCLIExample: Example extension for 3D Slicer that demonstrates how to make a Python script available as a CLI module

• Is a CLI script w/ an XML that has module metadata + examples

  • CLI module goes in the same examles menu
  • and a basic UI has been generated for it? Wow
  • Running it from the UI actually shows a progress bar and it doesn’t block the interface! Woohoo!

• Adding it added it to CMakeLists.xml: add_subdirectory(my_scripted_cli)

• python dev tools actually show the CLI used to run a CLI module!

  • the input arguments are filepaths, and slicer automatically generates and then deletes temporary files for this when running

• can’t debug it the usual way, I guess I’ll have to add the pydevd-pycharm things to the code

[VTK] /opt/3dslicer/bin/python-real /home/.../cli_model_service.py /tmp/Slicer-sh/IACHCI_vtkMRMLScalarVolumeNodeD.nrrd 1 /tmp/Slicer-sh/IACHCI_vtkMRMLScalarVolumeNodeF.nrrd 

(Auto-creating) GUI

• Slicer/Docs/developer_guide/parameter_nodes/gui_creation.md at main · Slicer/Slicer seems interesting

• CLI modules have an interface specified by the .xml file, and the interface is generated based on it

  • couldn’t find any documentation on how to write the XML: Documentation/Nightly/Developers/SlicerExecutionModel - Slicer Wiki

• Bits about the XML

  • channel is either input or output, and only for pics.
  • No parameter names, just their index

• If I read the volume as volume, it uses the .nrrd file extension by default — and I want .nii

  • Script repository deals with these settings, but I can’t use them from within a CLI module Script repository — 3D Slicer documentation

Auto-loading the module at startup

Module from console with arguments - Support - 3D Slicer Community

Slicer.exe --python-code "selectModule('SampleData'); getModuleGui('SampleData').setCategoryVisible('BuiltIn', False)"

File operations

Script repository — 3D Slicer documentation:

# Create a new directory where the scene will be saved into
import time
sceneSaveDirectory = slicer.app.temporaryPath + "/saved-scene-" + time.strftime("%Y%m%d-%H%M%S")
if not os.access(sceneSaveDirectory, os.F_OK):
  os.makedirs(sceneSaveDirectory)

Inspiration

• User_Manual.md · develop · OpenDose / SlicerOpenDose3D · GitLab
  • openDose3D is a cool example extension that shows many things
• SlicerTotalSegmentator/TotalSegmentator/TotalSegmentator.py at main · lassoan/SlicerTotalSegmentator

  		outputSegmentation.AddDefaultStorageNode()
  		outputSegmentation.GetStorageNode().SetFileName(output_segmentation_path)
  		outputSegmentation.GetStorageNode().ReadData(outputSegmentation)
  

Useful bits from documentation

• There’s a node type for tables
  • statistics become a table
  • also: many modules exist
    • Documentation/Nightly/Extensions/MeshStatistics - Slicer Wiki / DCBIA-OrthoLab/MeshStatisticsExtension
      • does a lot including creating a table GUI programmatically and CSV export etc.
• Colors: are not just colors, but class/label names as well

• Main links

  • Extensions — 3D Slicer documentation¹
  • Extensions Manager — 3D Slicer documentation²

• slicerrc.py is a thing! sample: 3D-Slicer-Scripts/.slicerrc.py at master · jzeyl/3D-Slicer-Scripts · GitHub

• Slicer custom application deployment to many computers - Development - 3D Slicer Community

  • SlicerQReads.exe --python-code "folder='S:\SlicerQREADS\TestImages'; import os; slicer.util.loadVolume(folder + '/' + os.listdir(folder)[0], {'singleFile': False})"
  • alternatively: Slicer.exe --python-script "/full/path/to/myscript.py" --no-splash --no-main-window³

• Slicer can run in Docker⁴

• Slicer can run in webbrowser⁵

• Slicer can run in jupyter⁵, including partially (e.g. only a single view etc!)

Slicer --help:

  --testing                                     Activate testing mode. It implies --disable-settings and --ignore-slicerrc. (default: false)
  --disable-python                              Disable python support. This is equivalent to build the application with Slicer_USE_PYTHONQT=OFF.
  --python-script                               Python script to execute after slicer loads.
  --python-code                                 Python code to execute after slicer loads.
  -c                                            Python code to execute after slicer loads. By default, no modules are loaded and Slicer exits afterward.
  --ignore-slicerrc                             Do not load the Slicer resource file (~/.slicerrc.py).
  --additional-module-path                      Additional module path to consider when searching for modules to load.
  --additional-module-paths                     List of additional module path to consider when searching for modules to load.

• Slicer modules — todo, CLI things with easy arguments

  • Documentation/Nightly/Developers/Modules - Slicer Wiki

• Slicer extensions/modules creation

  • Developing extension in 3D Slicer - Google Slides presentation
    • 
    • Sharing extensions there: TL;DR use slicer publishing thing to publish to own repo, then PR to slicer main extensions index repo (GitHub - Slicer/ExtensionsIndex: Slicer extensions index)
      • Sample JSON: ExtensionsIndex/SurfaceWrapSolidify.json at main · Slicer/ExtensionsIndex · GitHub
      • Pull requests · Slicer/ExtensionsIndex has code for existing recent extensions that may be useful

• Extension manager

  • Extensions Manager — 3D Slicer documentation ²
  • Can have different extension server URI!
  • Updating:
    • Extensions Manager — 3D Slicer documentation
    • TL;D/R extension manager can do that incl. automatically

• CI/CD exists for extensions published officially

• Misc:

  • Script repository — 3D Slicer documentation has sample uses of slicer CLI!
  • 3D-Slicer-Scripts/.slicerrc.py at master · jzeyl/3D-Slicer-Scripts · GitHub

• Script repository — 3D Slicer documentation has this:

  • This code snippet can be useful for sharing code in development without requiring a restart of Slicer.

  • gist.githubusercontent.com/pieper/a9c0ba57de3833c9f5aea68247bda597/raw/82d73d85340ffd205b01fc339b096a8607cfe02b/InstallSlicerModuleFromGit.py
  • TODO, does this mean updating extensions without restart?
  • EDIT: I can reload the code of the extension as-is

• Fazit:

  • extensions can be built (C++) or not (python)
  • enabling:
    • add to CLI path to “additional module paths” path to place where .py extension is locally
      • git pull etc.?
      • get a .zip from the repo and use the python code to get it from there and update
    • make our own extension manager repo thing and use automatic updating functionality

• Other bits

  • Making Slicer simpler by hiding many things: Simple 3D Slicer based image viewer
  • And hiding a module UI: Slicer/Modules/Scripted/DICOMPlugins/DICOMScalarVolumePlugin.py at 022a4f1a347ebb24509a50f6c8928ae26d3c5955 · Slicer/Slicer

inotifywait(1) - Linux man page

#!/bin/sh
while inotifywait --format '%:e %f' p.* *.bib; do
  sleep 1  # files get moved often
  echo "rsync!"
  rsync -avzirh --progress  p.* *.bib me@server.net:dir
done

boltctl monitor shows changes as they happens — nice when something breaks. I used to do a lot of watch boltctl list before in this exact scenario

(having a monitor subcommand in your thing says a lot about stuff, actually, — the best interpretation is that it’s written with developers in mind)

Very proud of this idea, haha :)

For absolute positioned elements one needs to “guess” the correct sizes etc., may not be trivial. Using a pixel ruler won’t help because the .absolute coords don’t map to the screen ones.

Solution:

.grid {
  // https://stackoverflow.com/questions/4191260/drawing-a-grid-using-css
  background-image:
    repeating-linear-gradient(lightblue 0 2px, transparent 2px 100%),
    repeating-linear-gradient(90deg, lightblue 0 2px, transparent 2px 100%);
  background-size: 50px 50px;
  //border: 2px solid red;
}

.smallgrid {
  background-image:
    repeating-linear-gradient(#ccc 0 1px, transparent 1px 100%),
    repeating-linear-gradient(90deg, #ccc 0 1px, transparent 1px 100%);
  background-size: 10px 10px;
  width: 100%;
  height: 100%;
}

::::: {.absolute left=0 right=0 top=0 bottom=0 .grid}
::: {.smallgrid}
:::
:::::

It’s not exactly aligned but close enough to judge the widths and heights involved!

Rectangle in the pic is:

:::: {.redrect .absolute right=200 top=130 width=200 height=100}
::::

(And the small grid can be omitted if not needed by removing the div in the middle. )

For bonus points, can be saved as a file and included if needed:

{\{< include ../_shared/smallgrid.qmd >}\}

ALSO:

// show the border of the slide
.reveal .slides {
  border: 1px dashed red;
}

DAMN.

Figures – Quarto:

:::: {layout="1,1,1],[1],[1,1,1" layout-valign="center"}

if it’s a layout I do get a valign thing. I just have to use layouts, not columns. Damn.

This doesn’t work on SVGs for me:

![](images/02/svgs/02_seqsiii.svg){.absolute top=0 left=0}

This does:

::: {.absolute top=0 left=0}
![](images/02/svgs/02_seqsiii.svg)
:::

SO.

One can literally generate a plot w/ graphviz, export as svg, open and edit in inkscape, save as optimized svg, paste into quarto, and manually add the correct class="fragment" data-fragment-index=4 bits to the relevant groups.

Insert with the usual {=html} thing.

Damn.

LeaderLine

almost works with quarto, except that line.position needs to be called on slide change — ergo likely won’t work in pdf export mode.

maurosilber/pyfragments: Animated figures for Quarto installs as-is

IT WORKS FOR PDF MODE TOO! And uses fragments

# | fig-width: 15
# | fig-height: 10
# | output: asis
import matplotlib.pyplot as plt
from pyfragments import AnimatedFigure

with AnimatedFigure() as ani:
# Data for plotting
  train_loss = [1.0, 0.6, 0.3, 0.2, 0.1]
  val_loss = [1.2, 0.8, 0.5, 0.4, 0.6]
  iterations = [1, 2, 3, 4, 5]

  plt.xlim(0, 6)
  plt.ylim(0, 1.5)

# Labels and legend
  plt.xlabel('Trainingsiterationen', fontsize=14)
  plt.ylabel('Loss', fontsize=14)
  plt.legend(loc='upper right', fontsize=12)

# Plot
  for x in range(len(train_loss)+1):
    with ani.fragment():
      plt.plot(iterations[:x], train_loss[:x], 'bo-', label='Training', linewidth=2)
      plt.plot(iterations[:x], val_loss[:x], 'go-', label='Validation', linewidth=2)

# Add a dashed vertical line in the middle
  mid_iteration = (iterations[0] + iterations[-1]) / 2

  with ani.fragment():
    plt.axvline(x=mid_iteration, color='r', linestyle='--', linewidth=1.5)

I couldn’t make a graph large enough to fill the screen, increasing fig-width didn’t help.

Solution: increase both fig-width and fig-height!

{dot}
//| fig-width: 12
//| fig-height: 9
digraph G {
    rankdir=LR;
	..

Execution Options – Quarto has figure options that lists the default sizes of figures based on output formats. Couldn’t find them because was looking in figures/graphviz etc. pages, not execution. …

For reveal slides, it’s 9 x 5.

Apparently it didn’t want to increase the 5 till I explicitly told it to, then width started increasing as well.

Both superscripts and subscripts

… are hard and you have to use a table.

digraph Neural_Network {
    rankdir=LR;
    ranksep=1.3;
    
    node [shape=circle, style=filled, fontcolor=white, fontsize=25, fillcolor="blue", color="black"];

    subgraph cluster_0 {
        node [fillcolor="#2c3e50", style="filled"];
        x1 [label=<
        <TABLE border="0" cellborder="0" cellspacing="0">
        <TR><TD rowspan="2" style="...">X</TD><TD style="...">1</TD></TR>
        <TR>                                  <TD style="...">1</TD></TR>
        </TABLE>>];
        x2 [label=<
        <TABLE border="0" cellborder="0" cellspacing="0">
        <TR><TD rowspan="2" style="...">X</TD><TD style="...">1</TD></TR>
        <TR>                                  <TD style="...">2</TD></TR>
        </TABLE>>];
    }

    sum [label=<∑<FONT color="yellow" point-size="10">(⎰)</FONT>>,  fillcolor="#27ae60", width=0.8, height=0.8, fixedsize=true];
    y [label=<y<sup>1</sup>>];
    
    edge [style=solid, color="#2c3e50"];
    x2 -> sum;
    x1 -> sum;

    edge [style=solid, color="#27ae60"];
    sum -> y;

    {rank=same; x1; x2;}
}

Commit messages starting with #14 whatever are awkward as # is the default comment in git rebase and friends.

git config core.commentchar ";"

fixes that for me.

For a one-time thing this works as well:

git -c core.commentChar="|" commit --amend

(escaping - Escape comment character (#) in git commit message - Stack Overflow)

Misc

• Papers
  • Comparison of Three 3D Segmentation Software Tools for Hip Surgical Planning - PMC: compares 3d slicer, Syngo, and Materialize¹
    • 
  • Comprehensive Review of 3D Segmentation Software Tools for MRI Usable for Pelvic Surgery Planning - PMC²: 12 software tools
    • really nice criteria, incl. extensibility etc.
    • 
    • 
    • TL;DR: ITK-SNAP most usable, Myrian Studio second but windows-only, 3d Slicer third but neat because easily extendable. Seg3d as also mentioned as aving very good usability
• Misc
  • https://github.com/neurolabusc/niivue-images sample nii images for evaluating nii viewers/editors

+ 3d Slicer

• Main file: 240715-1653 Notes on 3d-slicer, 240718-1530 First python 3d slicer interactions
• License:
  • About 3D Slicer — 3D Slicer documentation
  • custom license: BSD-style, broadly open source
    • no GPL/copyleft code allowed because it’d make it harder to mix slicer w/ private intellectual property
    • commercial use OK
    • 3d slicer based products — only some are listed, because one can use it and not disclose it and it’s fine
• Language: python
• extensible in python as well
  • has many cool extensions: @KitwareMedical/slicer-extensions-webapp
  • includes MONAILabel active learning extension plugin
  • a whole category for developer tools! @KitwareMedical/slicer-extensions-webapp
  • Extension Wizard — 3D Slicer documentation
    • https://extensions.slicer.org/catalog/Shape Analysis/32952/linux has extensions about stats
  • ai-assisted-annotation-client/slicer-plugin/README.md at master · NVIDIA/ai-assisted-annotation-client
• Very active community and frequent updates
• Downsides
  • J:
    • too specific for medical images
    • presets hard to customize (I agree!)
    • bad documentation (I think fixed now)
  • but: “Best of the free ones”
• Ha: https://github.com/NVIDIA/ai-assisted-annotation-client/blob/master/slicer-plugin/README.md

+ Syngo

• syngo.via OpenApps – Siemens Healthineers Deutschland
• Paid.
• Best according to ¹
• No easy way to test.

MeVisLab

• https://www.mevislab.de/
• Commercial and non-commercial version available
• Documentation:
  • Getting Started
• SDK for scripting exists
  • python QT bindings available
  • and generally feels expandable w/ python, shows python errors in the main screen for example, which is a good sign
• Official video tutorials exist, incl. for application development: MeVisLab - YouTube
• Download: a single .bin file that has to be run
• I don’t really like the website but the program itself feels nice!
• Nice pipeline visualization: and adding blocks — much less chaotic than 3d slicer (at first glance)
• overall good impression
• python scripting has a nice reference etc.

ITK-SNAP

• TODO
• Winner of ² due to its usability, does segmentation only but does it well
• available on AUR
• tutorial: TutorialSectionVolumes
• I really like it as first impression
• Has a distributed segmentation service already implemented!
  • ITK-SNAP Distributed Segmentation Services
  • 
  • ASHS Documentation - Overview sample of such a service
  • (can’t test it because pyqt6 present but not found)
• has volumes and statistics, maybe expandable
• BUT can’t find easy ways to add plugins

+ 3D Seg

• Open Source, segmentation-first
• for linux has to be compiled from source
• last meaningful commits from ~3 years ago: Commits · SCIInstitute/Seg3D, last release 2021 as well
• tried to build it, as ./build, failed because can’t find svn for
• Has detailed-ish developer documentation as videos! https://www.sci.utah.edu/cibc-software/seg3d.html
• Couldn’t build locally sadly, error in the middle of it
• Giving up

Myrian

• commercial, but non-commercial exists, and can be extended with plugins

NeuroMorph

• NeuroMorph ‐ EPFL
  • NeuroMorph | BIII
• Blender-based!

• NeuroMorph is a set of tools designed to import, analyze, and visualize mesh models in Blender. It has been developed specifically for the morphological analysis of 3D objects derived from serial electron microscopy images of brain tissue, but much of its functionality can be applied to any 3D mesh.

ilastic

• ilastik - ilastik
• integrat-able with fiji: ilastik/ilastik4ij: ImageJ plugins to run ilastik workflows and with everything else
• BioImage.IO integration as well, btw another model zoo

Fiji

• Fiji: ImageJ, with “Batteries Included”
• Legendary
• infinite plugins, very well known, downloadable as executable
• not too easy to use in my subjective opinion, menus inside menus, but…

Kitware glance

https://kitware.github.io/glance/doc/index.html

• web-based, extendable
• Kitware/glance: Kitware Glance is an open-source web application developed at Kitware for visualizing volumetric images, molecular structures, geometric objects, and point clouds. It is part of the ParaView platform and can serve as a foundation for building custom web-based visualization applications involving ITK.js and VTK.js.

Biaflows

• https://biaflows-sandbox.neubias.org/#/
• fully-online and can run models
• I don’t like the 3d viewer and I think it’s segmentation only, no easy way to generate/show stats

Why is the number of voxels different from the number of matrices? - Support - 3D Slicer Community

>>> raw = getNode('probe1_0000.nii.gz')
>>> slicer.util.arrayFromVolume(raw).shape
(180, 180, 500)
>>> 

doesn’t work for segmentations because not a volume.

Image dimensions can be seen from volume information And are identical in both segmentations and original

I found smoothing in view controllerS!

In Slice controllers there’s this arrow button that allows to set basically layers AND SMOOTHING per slice!

So everything is as expected wrt smoothing — there was none.

Zooming in slices w/ CTRL!

// ref: 240701-1752 NII MRI annotation tools

• 3D Slicer image computing platform | 3D Slicer
• DICOM seems primary format¹ but nifti is supported
• Installing from AUR

• Importing 2x .nii as directory worked as-is

• Volumes can do settings per file, incl.:

  • annotations threshold starts at 1

• Volume renderings gives 3d views?

• Annotations can be imported as annotations/segmentations through add data! Then they are parsed semantically

• One can segment N slices and it magically creates a 3d shape out of it!

• Exporting annos as nifti possible through “export” in Segmentation module ²(not Segment Editor!)

• https://www.youtube.com/watch?v=CxwxYtizdhE resampling volumes

• Diagrams

  • A Quarto tip a day - Diagrams

• This shows how to use fragments inside code/diagram segments: Fragment Layout Elements in Reveal.JS · quarto-dev/quarto-cli · Discussion #3685

• Non-quarto solutions:

  • Using code to generate graphs: GraphViz as figure when generated by R code · quarto-dev/quarto-cli · Discussion #5949
  • mapio/GraphvizAnim: A tool to create animated graph visualizations, based on graphviz.

Simplest way possible

• Multiple fragments one after the other with less and less invisible graph parts
  • style=invis: sum [label="∑", fillcolor="#27ae60", shape=circle, style=invis];¹

This makes everything invisible by default, uncommenting the lines makes the parts visible

digraph G {
    rankdir=LR;
    node [shape=circle, style=invis, fontcolor=white, fontsize=18, fillcolor="white", color="white"];
    edge [color="white", fontcolor="white"];
    graph [color="white"];
  
    
  subgraph cluster_1 {
    #label = "Inputs";
    // uncomment for visibility:
    node [fillcolor="#2c3e50", style="filled"]
    
    n0 [label=<1>];
    n1 [label=<x<SUB>1</SUB>>];
    n2 [label=<x<SUB>2</SUB>>];
    n3 [label=<x<SUB>n</SUB>>];

    { rank=same; n0; n1; n2; n3; }
  }
  
  subgraph arrows {
    // uncomment for visibility of arrows:
    edge [color="black", fontcolor="black"];
    node [style="filled", fillcolor="#27ae60"];
    
    
    n0 -> sum [label=<w<SUB>0</SUB>>];
    n1 -> sum [label=<w<SUB>1</SUB>>];
    n2 -> sum [label=<w<SUB>2</SUB>>];
    n3 -> sum [label=<w<SUB>n</SUB>>];
 
    sum [label="∑"];
  }
    
  subgraph end {
    // uncomment for visibility
    edge [color="black", fontcolor="black"];
    node [style="filled", fillcolor="#27ae60"];

    activation [label="∫", fillcolor="#f39c12", shape=circle];

    // Output node
    output [label="y", fillcolor="#8e44ad"];

    // Other edges
    sum -> activation;
    activation -> output;

   }

}

• The very best QR code generator is https://hovercode.com/
• DuckDuckGo can generate simple ones as well: qr https://www.eff.org/ at DuckDuckGo

Basics

Graphviz Online is a really nice viewer

Grouping

• You can set attributes globally or per subgraph
• edge [color="white"] makes all child edges white, unless overwritten

Labeling things

I figured out this way to label different ranks by using an invisible pseudo-graph: viewer

This is the chunk:

  subgraph labels {
    edge [fontcolor="black"];
    node [style="filled", fontcolor="black", fontsize=10];
      
    node1 [label="Inputs"];
    node2 [label="Weights"];
    node3 [label="Sum"];
    node4 [label="Activation\nfunction"];
    node5 [label="Output"];
    
    node1 -> node2 -> node3 -> node4 -> node5;
    
    {rank=same; node1;n0;n1; n2; n3;}

    
    {rank=same; node3;sum;}
  }

Note using ranks to align inputs/sum to the correct level; also the use of \n in the label.

Adding labels under nodes

• xlabel, and use HTML to make it a diff color than the main label. dot - Graphviz: Putting a Caption on a Node In Addition to a Label - Stack Overflow Graphviz: Distinct colors for xlabel and nodelabel - Stack Overflow

• To get images, rename the pptx to .zip¹ and just copy them from the archive
  • zipinfo lists everything

• ssine/pptx2md: a pptx to markdown converter does this automatically!

  • --qmd
  • Had to install deps manually
  • In LibreOffice could delete the footer and header in the master slide, giving me clean slides.
    • LibreOffice could delete header only, but not the footer — had to use MS Office for this

• // Ha: [Falk358/quarto_translator: A python project for translating quarto markdown files using the openai API](https://github.com/Falk358/quarto_translator+

  • Tested it on a small file, it actually works nicely

LaTeX

Colorful equations

How do I write LaTeX with colors in quarto presentations? · quarto-dev/quarto-cli · Discussion #4429

$$\begin{aligned}
\color{green}{\bf{Net Benefit}} = \frac{\color{green}{\bf{TP}}}{\text{N}} - \frac{\color{red}{\bf{FP}}}{\text{N}} * {\frac{{p_{t}}}{{1 - p_{t}}}}
\end{aligned}$$

$$\color{red}{w_0} + x_1*w_1 + x_2*w_2 + x_1*w_1$$  

It uses the usual names: LaTeX Color Definitions

Actually² these: MathJax-src/ts/input/tex/color/ColorConstants.ts at master · mathjax/MathJax-src

HA, it can even do usual RGB hex colors!

The useful to me are Cardinal/Carmine and Cerulean/Blue?..

Size of formulas

• Modifying the MathJax size · quarto-dev/quarto-cli · Discussion #2392 only global

• Mathjax supports things like \tiny{w_..}! (Changing mathjax’s font size - Stack Overflow)

Quarto align stuff right

Quarto: How to right justify text in PDF and HTML - Stack Overflow:

\`\`\`{=html}
<style>
.flushright {
   text-align: right;
}
</style>

::: {.flushright}

But actually,

::: {style="text-align: right;"}
$\small{E(W) = \frac{1}{n} \sum_{i=1}^{n} \mathcal{L} \left( \color{blue}{f(x^{(i)}; W)}, \color{red}{y_{true}^{(i)}} \right)}$
:::

Height and aspect ratios

Preview in a long/high browser window to see what the actual width of the presentation is. PDF export also uses the actual width

Misc

• ✕ aka MULTIPLICATION X is a really nice cross mark to use in random pictures.³
  • \times in latex
• ⮕ is a good arrow.
  • → is better (→)
  • more: HTML Unicode UTF-8

Context: in a paper, using PDF graphics is much better than e.g. PNG, because antialiasing (230628-1313 Saving plots matplotlib seaborn plotly PDF). Assume you need to crop one, and taking a PNG screenshot won’t cut it, because it would defeat the entire purpose.

LaTeX

TL;DR: in \includegraphics, trim=left bottom right top (in cm)

positioning - How can I crop included PDF documents? - TeX - LaTeX Stack Exchange:

includegraphics[clip, trim=0.5cm 11cm 0.5cm 11cm, width=1.00\textwidth]{PDFFILE.pdf}

BUT! If I’m ever in this situation, inkscape can really nicely open a PDF and export only the selected things, also as PDF.

Inkscape (much better)

1. When importing, “Replace text with closest available font”
   • e.g. OpenSans gets matched to Open Sans and it works out usually
   • the full list of fonts is shown in the import window
2. Select the part I want to crop
   1. or create a shape over it and select that shape or something
3. Export selection only as PDF; play with export settings as well if needed but for me the defaults worked this time

And either way

… test the results by downloading the PDF from overleaf and opening it locally, because antialiasing w/ imported vector pictures can be broken from within the preview window.

Used Apps/Dia - GNOME Wiki! all my life and love it, but

News! 2011-Dec-18: Version 0.97.2 has been released.

Depends on python 2.7 and is very problematic to install now. I’ll miss it.

I hear https://draw.io is a good alternative.

I’ll add a list of options here next time I need this.

(But for older dia files I’ll have to edit maybe later: it could be installed on a Windows machine and export a broken .SVG which I could then edit in Inkscape.)

Closing the window with the kitty terminal through qtile/OS means often asks you whether you really want to, as sth is running there.

Kitty has Ctrl+Shift+W which closes the terminal no questions asked.

For yay, the cache is in

$HOME/.cache/yay/

If the package is there, then: sudo pacman -U ./quarto-cli-1.4.555-1-x86_64.pkg.tar.zst

Then pin the package in /etc/pacman.conf:

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
IgnorePkg   = quarto-cli
#IgnoreGroup =

Then yay -Syu will ignore it as well:

:: Synchronizing package databases...
 endeavouros is up to date
 core is up to date
 extra is up to date
 multilib is up to date
:: Searching AUR for updates...
 -> quarto-cli: ignoring package upgrade (1.4.555-1 => 1.5.52-1)
:: Searching databases for updates...
 there is nothing to do

Refs: How do you downgrade an AUR package? : r/archlinux

For not-AUR, there’s the downgrade command: archlinux-downgrade/downgrade: Downgrade packages in Arch Linux

References

• Refs:
  • Revealjs Options – Quarto¹
  • Advanced Reveal – Quarto
  • Revealjs – Quarto
• Previously:
  • 240424-1923 Presentations with Quarto and Reveal.js has my list of tricks
• Code
  • Quarto reveal.scss: quarto-cli/src/resources/formats/revealjs/reveal/css/reveal.scss at 303c4bc9f52ea0b0c36c5eb5dfbade9cece2b100 · quarto-dev/quarto-cli
  • Quarto Reveal footer CSS: quarto-cli/src/resources/formats/revealjs/plugins/support/footer.css at main · quarto-dev/quarto-cli

Misc

Better preview at a specific port, handy for restarting after editing CSS. Similar to quarto project frontmatter preview: port: 4444 browser: false.

quarto preview slides.qmd --port 4444 --no-browser

Supported by default

In frontmatter:

• logo is in bottom-right+
  • footer-logo-link
• footer for all slides
  • overwrite with div of class footer
• Likely relevant for me: Numbering reveajs options for dynamic bits, shift-heading-level-by, number-offset, number-section, number-depth — will touch if I need this.

Headers

• Extension: shafayetShafee/reveal-header: A Quarto filter extension that helps to add header text and header logo in all pages of RevealJs slide²
  • Has absolutely awesome documentation!

In front matter:

• header: one header text for all presentation
• title-as-header,subtitle-as-header: if true, place the presentation frontmatter’s title/subtitle as header if one not provided (overwriting the header value)
  • use-case: I same repeating text everywhere w/o specifying it every time.
• hide-from-titleSlide: all/text/logo to hide it from title slide
• sc-sb-title: if true, h1/h2 slide titles will appear in the slide header automatically when slide-level is 2 or 3

Divs with classes:

• .header: slide-specific header

• Excellent example from its docu of dynamic per-(sub-)section headers:

  • code: reveal-header/example_all.qmd at main · shafayetShafee/reveal-header
  • demo: Quarto Presentations

Left/right/center blocks of text in header

³

• If one uses section/subsection titles, they go in the left third and right third of the header, with the normal header text in the middle. This is neat to have in general, w/o the section/subsection titles.
  • sc-title, on the left, is section title
  • sb-title, on the right, is subsection title
• How do I add arbitrary text there? And what would be a good interface for it?

The right way — extension code

• I could look at the extension, maybe fork it, and find a way to put text in these divs
• Relevant code:
  • This JS populates them in each slide: reveal-header/_extensions/reveal-header/resources/js/sc_sb_title.js at main · shafayetShafee/reveal-header
  • add_header js: reveal-header/_extensions/reveal-header/resources/js/add_header.js at main · shafayetShafee/reveal-header
  • Main lua: reveal-header/_extensions/reveal-header/reveal-header.lua at main · shafayetShafee/reveal-header

Ugly CSS hack

• (Learn sass in Y Minutes)
• CSS for the header bits: reveal-header/_extensions/reveal-header/resources/css/add_header.css at main · shafayetShafee/reveal-header

Since .s[c|b]-title is present always, text added that way will be present on the title slide regardless of settings. … and — adding text to a presentation through CSS is, well, ..

.reveal-header .sc-title {
  background-color: red;

	&::before {
		content: 'sc-title header content';
  }

}

Slightly better ugly hack: main header text split in three, with two aligned l/r correspondingly.

.header-right {
  // text-align: left;
  float: right;
  background-color: pink;
  display: inline-block;
}

.header-left {
  // text-align: left;
  float: left;
  background-color: yellow;
  display: inline-block;
}

::: header 
[left]{.header-left} and not left [and right]{.header-right}
:::

Result:

Extended ugly SCSS hack

Improved the above to support both footer and headers I have a hammer and everything is a nail SCSS can do mixins let’s use them

// L/R margins of the footer — if logo is wider than this, it may overlap right footer text.
// By default, logo max-height is 2.2em, width auto based on this.
$footer-margin: 5em;

// left or right column bits
%hfcol {
  display: inline-block;
}


/* L/R columns in header */
.reveal .reveal-header .rightcol {
  @extend %hfcol;
  float: right;
}

.reveal .reveal-header .leftcol {
  @extend %hfcol;
  float: left;
}

/* L/R columns in footer */
.reveal .footer .leftcol {
  @extend %hfcol;
  float: left;
  margin-left: $footer-margin;
}

.reveal .footer .rightcol {
  @extend %hfcol;
  float: right;
  margin-right: $footer-margin;
}

Usage:

## I am a slide  with custom headers

And With content 

::: footer 
[Left footer]{.leftcol} Center footer [Right footer]{.rightcol}
:::

::: header 
[Left header]{.leftcol} Center header [Right header]{.rightcol}
:::

Frontmatter usage works only for footer, likely header doesn’t support markup.