I know enough German to judge but not enough to write grammatically myself, and this is a pattern I don’t want to forget.

This is SO MUCH BETTER than anything else I’ve used for this! For very short snippets it’s golden

For the main file, see: 240116-1701 LaTeX best practices

Camera-ready checklist

• Enable reviewing mode in Overleaf
• Change package options [review/final] etc.
• Authors emails, names, order correct?
• Acknowledgements present and uncommented?
• Anonimity: uncensor names+URIs
• Are acknowledgements, references, appendices a) allowed, b) part of page limit?
• COMPRESSION: reconsider all hacks if you got more space for the camera-ready version
  • \setlength{\itemsep}{0pt} \setlength{\parskip}{0pt} and friends
  • \setlength{\belowcaptionskip}{0} \setlength{\abovecaptionskip}{10pt}¹ <- default values
  • \looseness-1
  • Any paragraphs to un-merge for readability?
• Go through pre-flight checklist below
• Manually check ALL review-mode changes at the end!

Pre-flight checklist

• Ask LLMs to proofread! THEN do the steps below.
• Acknowledgements, references, appendices: a) allowed, b) count towards page limit?
  • In which order?
• Anonymity: acknowledgements, again!
• Search for TODOs in the text!
• Consistency:
  • When to use single/double quotes (quotes), when emphasis (foreign languages?)
  • terminology?
    • (including in captions and inside figures (labels, titles!))
  • Table headers’ variables and names are easy to forget to update!
• Tables
  • Did any footnote markers get deleted when pasting new data in tables?
  • Adding \bottomrules helps if distance between table and caption too small
• Titles
  • Title Case where needed (paper title + sections)
  • Break the paper title in some pretty way
    • \mbox to avoid breaks, \- within word to mark break words
• Citations+references
  • before periods and footnotes
  • always w/ NBSP
  • Double-check each reference, including inside table/figure captions!
  • citep=cite, citet if subject, citealp in parentheses
  • Don’t forget multiple citations exist [1,3]!
• Commands/macros
  • Search for \TODO and any custom commands/macros you created
  • Check the spacing after any custom macros
• Typography bits
  • 231206-1501 Hyphens vs dashes vs en-dash em-dash minus etc
  • Smart Quotes: Either `x’ and ``x’’ or \enquote{} everywhere
  • Large Numbers: Write large numbers as 54{,}000.99
  • ~ is \textasciitilde: check any estimate numbers (~50%)!
  • % is \%; check all percentages!
  • \ is \textbackslash
  • underscores, subscripts, superscripts are dangerous
  • Avoid footnotes after numbers
  • does minted caption use the same font as the rest?
  • -,--,--- are the safest — look and remove pasted UTF8 ones
• NBSP (search and manually check each!)
  • Citations, \references (Figure~\ref{fig:somefig})
  • TODO
• Look at logs!

Things that went wrong content-wise in the past

• Read through all analysis and tables, make sure the data makes sense, the analysis/results match the numbers in the tables, the plots and the tables match to each other.
• Make sure whatever you used to generate plots and tables used the exact same data!

I wrote this debugging some metrics, bad code but I don’t want it to die. Shows formulas for Precision/Recall/F-Score based on number of TP TN FP FN.

Depends only on rich, runnable w/ uv through 250128-2149 Using uv as shebang line.

Usage: uv run prftset.py tp tn fp fn

# /// script
# dependencies = [
#   "rich",
#   "pdbpp", # ви цього варті
# ]
# ///

from rich import print
from rich.layout import Layout
from rich.table import Table
import argparse


def get_scores(tp, tn, fp, fn) -> dict[str, int | float | str | None]:
    """Calculate precision, recall, and F1 score from true positives, true negatives, false positives, and false negatives."""

    precision = tp / (tp + fp) if (tp + fp) > 0 else None
    recall = tp / (tp + fn) if (tp + fn) > 0 else None
    f1_score = (
        2 * (precision * recall) / (precision + recall)
        if precision is not None and recall is not None and (precision + recall) > 0
        else None
    )
    accuracy = (tp + tn) / (tp + tn + fp + fn) if (tp + tn + fp + fn) > 0 else None

    f_precision = float(f"{precision:.3f}") if precision is not None else "N/A"
    f_recall = float(f"{recall:.3f}") if recall is not None else "N/A"
    f_f1_score = float(f"{f1_score:.3f}") if f1_score is not None else "N/A"

    total_ds = tp + tn + fp + fn

    res = {
        "TP": tp,
        "TN": tn,
        "FP": fp,
        "FN": fn,
        "total": total_ds,
        "P ": f_precision,
        "R ": f_recall,
        "F ": f_f1_score,
        "A ": float(f"{accuracy:.3f}") if accuracy is not None else "N/A",
        # "A ": accuracy,
    }

    return res


def use_table(tp, tn, fp, fn, precision, recall, accuracy):
    total = tp + tn + fp + fn
    total_positive = tp + fn
    total_negative = tn + fp

    table_tn = Table()
    table_tn.add_column("TP", justify="center", style="green")
    table_tn.add_column("TN", justify="center", style="green underline italic")
    table_tn.add_column("FP", justify="center", style="red")
    table_tn.add_column("FN", justify="center", style="red underline italic")
    table_tn.add_column("sum/+/-", justify="right", style="cyan")
    table_tn.add_row(
        f"[bold green]{str(tp)}[/bold green]",
        f"[bold green]{str(tn)}[/bold green]",
        f"[bold red]{str(fp)}[/bold red]",
        f"[bold red]{str(fn)}[/bold red]",
        f"[bold cyan]{str(total)}[/bold cyan]="
        + f"[bold green]{str(total_positive)}[/bold green]+"
        + f"[bold red]{str(total_negative)}[/bold red]",
    )

    table = Table()

    table.add_column("", justify="right", style="cyan", no_wrap=True)
    table.add_column("Formula", justify="center", style="magenta")
    table.add_column("=>", justify="center", style="magenta")
    table.add_column("Value", justify="center", style="magenta")

    table.add_row(
        "",
        f"TP=[bold green]{str(tp)}[/bold green]",
        f"[bold yellow]{str(tp)}[/bold yellow]",
        "",
    )
    table.add_row(
        "Pre",
        "_" * (len(str(tp)) + len(str(fp)) + 6),
        "___",
        f"P=[bold yellow]{precision}[/bold yellow]",
    )
    table.add_row(
        "",
        f"TP=[bold green]{str(tp)}[/bold green] + FP=[bold red underline italic]{str(fp)}[/bold red underline italic]",
        f"[bold yellow]{str((tp + fp))}[/bold yellow]",
        "",
    )

    table.add_section()

    table.add_row(
        "",
        f"TP=[bold green]{str(tp)}[/bold green]",
        f"[bold yellow]{str(tp)}[/bold yellow]",
        "",
    )
    table.add_row("Rec", "______________", "", f"R=[bold yellow]{recall}[/bold yellow]")
    table.add_row(
        "",
        f"TP=[bold green]{str(tp)}[/bold green] + FN=[bold red underline italic]{str(fn)}[/bold red underline italic]",
        f"[bold yellow]{str((tp + fn))}[/bold yellow]",
        "",
    )

    table.add_section()
    # Accuracy
    table.add_row(
        "",
        f"TP=[bold green]{str(tp)}[/bold green] + TN=[bold green underline italic]{str(tn)}[/bold green underline italic]",
        f"[bold yellow]{str((tp + tn))}[/bold yellow]",
        "",
    )
    table.add_row(
        "Acc",
        "______________________________",
        "",
        f"A=[bold yellow]{accuracy}[/bold yellow]",
    )
    table.add_row(
        "",
        f"TP=[bold green]{str(tp)}[/bold green] + TN=[bold green underline italic]{str(tn)}[/bold green underline italic] + FP=[bold red]{str(fp)}[/bold red] + FN=[bold red underline italic]{str(fn)}[/bold red underline italic]",
        f"[bold yellow]{str((tp + tn + fp + fn))}[/bold yellow]",
        "",
    )

    print(table)
    print(table_tn)


def pretty_scores(scores: dict[str, int | float | str | None], draw=True):
    tp, tn, fp, fn = scores["TP"], scores["TN"], scores["FP"], scores["FN"]
    precision, recall, f1_score = scores["P "], scores["R "], scores["F "]
    accuracy = scores["A "]
    # print( f"| TP: {scores['TP']}, TN: {scores['TN']}, FP: {scores['FP']}, FN: {scores['FN']} | ")
    # print(
    #     f"| Precision: {scores['P ']}, Recall: {scores['R ']}, F1 Score: {scores['F ']} |"
    # )
    print(
        f"Precision: [bold cyan]P[/bold cyan] = [bold green]{tp}[/bold green] / ([bold green]{tp}[/bold green] + [bold red]{fp}[/bold red]) = [bold yellow]{precision}[/bold yellow]"
    )
    print(
        f"Recall: [bold cyan]R[/bold cyan] = [bold green]{tp}[/bold green] / ([bold green]{tp}[/bold green] + [bold red]{fn}[/bold red]) = [bold yellow]{recall}[/bold yellow]"
    )
    print(
        f"F1 Score: [bold cyan]F1[/bold cyan] = 2 * ([bold yellow]{precision}[/bold yellow] * [bold yellow]{recall}[/bold yellow]) / ([bold yellow]{precision}[/bold yellow] + [bold yellow]{recall}[/bold yellow]) = [bold magenta]{f1_score}[/bold magenta]"
    )
    print(
        f"Accuracy: [bold cyan]A[/bold cyan] = ([bold green]{tp}[/bold green] + [bold green]{tn}[/bold green]) / ([bold green]{tp}[/bold green] + [bold green]{tn}[/bold green] + [bold red]{fp}[/bold red] + [bold red]{fn}[/bold red]) = [bold yellow]{scores['A ']}[/bold yellow]"
    )

    # use_layout(tp, tn, fp, fn, precision, recall)
    use_table(tp, tn, fp, fn, precision, recall, accuracy)


def main():
    args = parse_args()
    # Example usage
    # tp = 100
    # tn = 50
    # fp = 10
    # fn = 5
    tp = args.tp
    tn = args.tn
    fp = args.fp
    fn = args.fn

    scores = get_scores(tp, tn, fp, fn)
    pretty_scores(scores)


def parse_args():
    parser = argparse.ArgumentParser(
        description="Calculate precision, recall, and F1 score."
    )
    # use positional arguments for tp, tn, fp, fn
    parser.add_argument("tp", type=int, help="True Positives", default=0)
    parser.add_argument("tn", type=int, help="True Negatives", default=0)
    parser.add_argument("fp", type=int, help="False Positives", default=0)
    parser.add_argument("fn", type=int, help="False Negatives", default=0)
    # parser.add_argument("--tp", type=int, required=True, help="True Positives")
    # parser.add_argument("--tn", type=int, required=True, help="True Negatives")
    # parser.add_argument("--fp", type=int, required=True, help="False Positives")
    # parser.add_argument("--fn", type=int, required=True, help="False Negatives")
    return parser.parse_args()


if __name__ == "__main__":
    main()

• To force-kill pods w/o breaking k8s: kubectl delete pod <> --now

Ty KM (https://github.com/kkmarv) for the following, quoting verbatim:

The expected behaviour of kubectl delete heavily depends on its arguments. There are 4 possible but wrongly documented options. I’ve tried to compile different sources into this list:

• kubectl delete pod <>  will first send a SIGTERM, then a SIGKILL after X seconds (depends on the gracePeriods set by your pod)
• kubectl delete pod <> --grace-period=X (X>0)  will first send a SIGTERM, then a SIGKILL after X secs
• kubectl delete pod <> --now is equivalent to kubectl delete pod <> --grace-period=1

There’s also --force
Where things are possible to go wrong. It immediately deletes the pod from etcd, which will NOT kill the pod. It will too wait for the grace period before sending a SIGKILL. But even then it is not guaranteed that resources will be freed. There is even a warning in kubectl delete --help  about this. Only use this when there is no other way to kill a pod.

The force flag also allows to specify --grace-period=0 which will simply default to the grace period defined by the pod, effectively doing nothing since this is also the default behaviour.

Settings Management - Pydantic Validation is nice but doesn’t cover everything.

Passing None through CLI

• Settings Management - Pydantic Validation
• None by default b/c cli_avoid_json is the default,[]

 it will default to “null” if cli_avoid_json is False, and “None” if cli_avoid_json is True .

Fun fact, works inside complex nested JSON flags as well.

Passing complex JSON-like fields incl. tuples

# Assume this
class MySettingsClass(BaseSettings):
    I_am_a_list_of_tuples: list[tuple[str, str, str | None]] = Field(
        default=FILE_PAIRS, validation_alias=AliasChoices("fp", "file_pairs")
    )
	
FILE_PAIRS = [
    (
        "infer_doc_simplecriteria/pred_doc_simplecriteria.json",
        "merge_gold_simplecriteria/gold_simplecriteria.json",
    ),
    (
        "file2-1",
        "file2-2",
    ),
]

# Passing as a list of lists parses as a tuple
# if done wrong the tuple is parsed as a list of strings
uv run thing --I-am-a-list-of-tuples \
'"merge_gold_simplecriteria/gold_simplecriteria.json",  "infer_doc_simplecriteria/pred_doc_simplecriteria.json"'

TL;DR pydanticmodel.model_dump(mode=>"json")

Serialization - Pydantic Validation

To dump a Pydantic model to a jsonable dict (that later might be part of another dict that will be dumped to disc through json.dump[s](..)):

settings.model_dump()
>>>
{'ds_pred_path': PosixPath('/home/sh/whatever')}
# json.dump...
json.dumps(settings.model_dump())
>>> TypeError: Object of type PosixPath is not JSON serializable

json.dumps(settings.model_dump(mode="json"))
# works!

Shorthand object contsuction

I kept googling for the syntax but could never find it again.

// long: 
'{user: .user, title: .titles}'

// shorthand when key name matches original key name (user==.user)
'{user, title}'
// mix 
'{user, title_name: .title}'

From the man page:

You can use this to select particular fields of an object: if the input is an object with “user”, “title”, “id”, and “content” fields and you just want “user” and “title”, you can write
{user: .user, title: .title}
Because that is so common, there´s a shortcut syntax for it: {user, title}.

The long version allows to operate on they keys:

{id: .idx, metadata: .metadata.original_idx}

curl 127.0.0.1:8000/v1/chat/completions -s \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/data_pvc2/cache/models--CohereLabs--c4ai-command-r-08-2024/snapshots/96b61ca90ba9a25548d3d4bf68e1938b13506852/",
    "messages": [
      {
        "role": "user",
        "content": "Generate a short analysis in the given JSON schema."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "analysis_schema",
        "schema": {
          "type": "object",
          "properties": {
            "summary": { "type": "string" },
            "importance_score": { "type": "number" },
            "key_points": {
              "type": "array",
              "items": { "type": "string" }
            }
          },
          "required": ["summary", "importance_score", "key_points"],
          "additionalProperties": false
        },
        "strict": true
      }
    }
  }' | jq . > /tmp/out

Sample otuput:

{
  "id": "chatcmpl-9c3932339c7d4e1bace32c24ef6dbada",
  "object": "chat.completion",
  "created": 1770201387,
  "model": "/data_pvc2/cache/models--CohereLabs--c4ai-command-r-08-2024/snapshots/96b61ca90ba9a25548d3d4bf68e1938b13506852/",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": null,
        "content": "{\n    \"summary\": \"This analysis focuses on the impact of marketing strategies on customer engagement and sales performance for a hypothetical e-commerce company.\",\n    \"importance_score\": 0.85,\n    \"key_points\": [\n        \"Implemented an email marketing campaign with personalized recommendations based on past purchases: Increased open rates by 15% and contributed to a 12% rise in average order value.\",\n        \"Launched an influencer marketing program on social media: Achieved a 20% growth in brand awareness and drove a significant rise in website traffic from social platforms.\",\n        \"Optimised product listings through enhanced SEO techniques: Boosted organic visibility, resulting in a 25% increase in search-driven purchases.\",\n        \"Implemented an automated email nurturing funnel for abandoned carts: Improved cart recovery rate by 10% and increased overall conversions.\",\n        \"Analyzed customer reviews for product feedback and sentiment: Used insights to refine product strategies and enhance customer satisfaction.\"\n    ]\n}",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 16,
    "total_tokens": 219,
    "completion_tokens": 203,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null
}

To pretty-print the content it returned:

jq -r .choices[0].message.content /tmp/out | jq .

Simpler OpenAI-compatible JSON mode test

curl 127.0.0.1:8000/v1/chat/completions -s \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/data_pvc2/cache/models--CohereLabs--c4ai-command-r-08-2024/snapshots/96b61ca90ba9a25548d3d4bf68e1938b13506852/",
    "messages": [
      {
        "role": "user",
        "content": "Return a JSON object with keys summary, importance_score (number), key_points (string array)."
      }
    ],
    "response_format": { "type": "json_object" }
  }'

Bonus: literature

• Structured output Considered Harmful(tm)
  • [2408.02442] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models
  • also e.g. Structured outputs create false confidence | Hacker News
• Some people STRONGLY disagree
  • Say What You Mean: A Response to ‘Let Me Speak Freely’

security - How do I recursively shred an entire directory tree? - Unix & Linux Stack Exchange:

find <directory_name> -depth -type f -exec shred -v -n 1 -z -u {} \;

TIL: Indico - Home, allegedly used by ScaDs and CERN, open sourcer

The effortless open-source tool for event organisation, archival and collaboration

Looks really really nice and professional

Resources

• Label Studio Enterprise — Data Labeling & Annotation Tool Interactive Demo
  • I always look for it, it shows things like scrollable fields, columns, the building blocks basically. Especially the advanced config templates.

Template bits

Adding sample data to template

(Shown somewhere in examples d but I always have to look for it)

<Header value="xxx" />

<!-- {
"data": {
  "index": 4,
  }
} -->

Since spaces don’t matter, this is even better for copypasting new versions:

<Header value="xxx" />

<!-- { "data": 

{
  "index": 4,
}
  
} -->

toName

Applying stuff to multiple names: comma-separate them ¹ EDIT: this doesn’t result in an error but quietly drops annotations and makes them not-editable, TODO bug report!

<Header value="Which text is better?" size="6" />
<Choices ... toName="name_a,name_b">

Choices

• Choices by radio button instead of checkmarks for single choice: `

  • TL;DR always use this, I hate that the checkmark design language is going away

• Inline choices: layout="inline"

• Value is shown in the screen, alias can override that in json output:

  • <Choice alias="1" value="Completely mismatched tone/content"/> will be 1 in json `

Formatting

Formatting text with info — couldn’t find anything better than this abomination:

<View style="display: flex;">
	<Header value="How well is this advertisement tailored for group '" size="6" />
	<Header value="$target_group" size="6" />
	<Header value="'?" size="6" />
</View>

Or:

  <View style="display: flex;">    
    <Text name="l_obj_id" value="Object ID:" /> <Text name="t_object_id" value="$object_id"/>
    <Text name="l_target_group" value="| Target group:" />  <Text name="t_target_group" value="$target_group"/>
	<Text name="l_style" value="| Style:" /> <Text name="sty" value="$text_style"/> 
  </View>

Style/layout

Styling tables

The <Table> tag generates a table with headers based on dict with key/value pairs.

This compresses it:

  <Style>
    .ant-table-cell {
    	padding: 0;
    	font-size: 0.75rem;
    	line-height: 1; /* Removes vertical space added by line height */
        }
	.ant-table-tbody td {
    	padding: 10px !important; 
    }
	
	/* And this sets the column widths */
    .ant-table-thead th:first-child,
    .ant-table-tbody tr td:first-child {
    	width: 40%;
    }
    .ant-table-thead th:last-child,
    .ant-table-tbody tr td:last-child {
		width: 60%;
    }
  </Style>

Generating tables as HTML

The native tag has very few options for config, in most cases rolling our own is a good idea. This quick undocumented ugly function generates a label-studio-like table structure (except it’s missing missing <thead> that I don’t need) from a key/value dictionary.

def poi_to_html_table(poi: dict[str, str]) -> str:
    DIVS_OPEN = '<div class="ant-table ant-table-bordered"><div class="ant-table-container"> <div class="ant-table-content">'
    TABLE_OPEN = '<table style="table-layout: auto;">'
    TBODY_OPEN = '<tbody class="ant-table-tbody">'
    TR_OPEN = '<tr class="ant-table-row">'
    TD_OPEN = '<td class="ant-table-cell">'

    TABLE_DIVS_CLOSE = "</tbody></table></div></div></div>"

    res = f"{DIVS_OPEN}{TABLE_OPEN}{TBODY_OPEN}"
    for k, v in poi.items():
        res += f"{TR_OPEN}{TD_OPEN}{k}</td>{TD_OPEN}{v}</td></tr>"
    res += TABLE_DIVS_CLOSE
    return res

TO STYLE THEM, SEE BELOW

Styling HTML by including it inline

<HyperText name="poi_table" value="$poi_html" inline="true" />

If you want to generate HTML and apply styles to it, the HyperText should be inline, otherwise it’ll be an iframe. Above, $poi_html is a table that then gets styled to look like native tables through .ant-table-... classes, but the style lives in the template, not in $poi_html.

Scrollable areas

Very very handy for texts of unknown length, especially inside columns:

<View style="height: 500px; overflow: auto;">
	<Text name="text" value="$text" />
</View>

EDIT: max-height for multiple stacked windows is golden!

Making scrollbars visible

Don’t ask how or why, but this works as of 1.21.0. I don’t know enough tailwind to say why does the approach from this fiddle with !important fails. Or why does the background of the scrollbar gets set always on chrome, but only on hover for firefox.

.scroll {
	scrollbar-color: crimson blue !important; /* scrollbar, background */ 
	scrollbar-width: auto !important; /* Use OS default instead of thin, only Chrome */
}

Related: scrollbar-color - CSS | MDN, Scrollbar Styling with Tailwind and daisyUI - Scott Spence

Multi-column layouts

<View style="display: grid; grid-template: auto/1fr 1fr; column-gap: 1em">`

(or)

<Style>
	.threecols {display: grid; grid-template: auto/1fr 1fr 1fr;}
	.twocols {display: grid; grid-template: auto/1fr 1fr;}    
</Style>

Patterns

  <Choices name="xxxx"  choice="single-radio" toName="obj_desc_a" layout="inline">
    <View style="display: flex;">
      <Header size="6" value="Bad" />
      <Choice value="1" />
      <Choice value="2" />
      <Choice value="3" />
      <Choice value="4" />
      <Choice value="5" />
      <Header size="6" value="Good" />
     </View>
  </Choices>

(Preserving l/r width)

  	<View style="display: grid; grid-template: auto/300px 300px 300px; place-items: center;">
      <Header size="6" value="Bad" />
      <Rating name="444"  toName="obj_desc_a" maxRating="5" />
      <Header size="6" value="Good" />
	</View>

You can make the options short and place additional info in hints!

<Choice alias="3" value="Average coverage" hint="covers basics but misses some details"/>

Misc

• Disabling template/layout hotkeys (the ones on the options):
  • disable once in annotation settings, works across all projects
  • Set explicitly to nothing
• For tab/indent formatting when in the template editor, ctrl-(backspace/delete) helps

CSS etc. links

• Aligning items in a flex container - CSS | MDN

TIL, mentioned by HC&BB: Rclone “is a command-line program to manage files on cloud storage”.

S3, dropbox, onedrive, [next|own]cloud, sftp, synology, and ~70 more storage backends i’ve never heard about.

Previously-ish: 250121-1602 Kubernetes copying files with rsync and kubectl without ssh access

Previously: 231018-1924 Speedtest-cli and friends

TIL about:

• Fast.com (to netflix’s servers)
• speed.cloudflare.com with many details

I especially like fast.com, I’ll remember the URI forever.

Refs:

• Adding a local folder as storage backend: Label Studio Documentation — Cloud and External Storage Integration
• Input .json tasks format: Label Studio Documentation — Import Data into Label Studio

The steps are:

1. Create a local directory that will contain the files. Let’s assume /home/sh/w/t/labelstudiodata
2. Create a subfolder there to keep your images, e.g. pics, making the full path /home/sh/w/t/labelstudiodata/pics
3. Create your input data json.

[
  {
    "data": {
      "ref_id": 1,
      "image": "/data/local-files/?d=pics/cat1.png",
      "image2": "/data/local-files/?d=pics/cat2.png",
      "image3": "/data/local-files/?d=pics/cat3.png"
    }
  },
  {
    "data": {
      "ref_id": 2,
      "image": "/data/local-files/?d=pics/dhl1.png",
      "image2": "/data/local-files/?d=pics/dhl2.png",
      "image3": "/data/local-files/?d=pics/dhl3.png"
    }
  }
]

In the data, the paths are /data/local-files/?d=pics/cat1.png — start with /data/local-files/?d=, then the subdir, then the path to the file itself (here it’s flat: cat3.jpg)

4. Start label-studio thus:

LABEL_STUDIO_LOCAL_FILES_SERVING_ENABLED=true LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT=/home/sh/w/t/labelstudiodata label-studio

LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT should point to your folder, WITHOUT the subfolder (no pics/) , and be absolute.

5. Add your directory as storage backend.

Create your project as usual, and open its settings. The absolute local path is the SUBFOLDER of the document root:

“Treat every bucket as source” should be unchecked — in the documentation, they describe it differently from the screenshots, but it’s equivalent:

• 8. Import method - select “Tasks” (because you will specify file references inside your JSON task definitions)

“Check connection” should tell you if everything’s OK.

6. DON’T sync the storage. Import the json you created from project import.

You should see your tasks.

7. Bonus sample template for the above: []

<View>  <View style="display: grid;  grid-template-columns: 1fr 1fr 1fr; max-height: 300px; width: 900px">  <Image name="image1" value="$image"/>  
  <Image name="image2" value="$image2"/>  
  <Image name="image3" value="$image3"/>  </View>  
  <Choices name="choice2" toName="image2">  
    <Choice value="Adult content"/>  
    <Choice value="Weapons"/>  
    <Choice value="Violence"/>  
  </Choices>  
</View>

[Tried it, realized that it’ll replace 80% of my use-cases of jupyter / jq etc., and improve viewing of random csv/json files as well!

(Previously: 250902-1905 Jless as less for jq json data and csvlens)

• Parentheses are used for grouping: $somelist | where ($it in (open somefile.json)) | length

Config in ~/.config/nushell/config.nu, editable by config nu, for now:

$env.config.edit_mode = 'vi'
$env.config.buffer_editor = "nvim"  # `config nu`

alias vim = nvim
alias v = nvim
alias g = git
alias k = kubectl
alias c = clear
alias l = ls

alias o = ^open

[]

allenai/olmocr: Toolkit for linearizing PDFs for LLM datasets/training

Online demo: https://olmocr.allenai.org/

• can be exposed through eg vllm
• really cool results on messy docs

Tokenizing

curl -X 'POST' \
  'http://localhost:8001/tokenize' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "mistralai/Mistral-Small-24B-Instruct-2501",
  "prompt": "my prompt",
  "add_special_tokens": true,
  "additionalProp1": {}
}'

• Using uv with Jupyter | uv

Creating jupyterlab environments w/ uv

# Create a new uv environment
uv init --name bbk-presentation --bare
# Add ipykernel
uv add --dev ipykernel
# Install the kernel in jupyterlab
uv run ipython kernel install --user --env VIRTUAL_ENV $(pwd)/.venv --name=bbk-paper

# Add whatever you need to the environment
uv add pandas seaborn

# Support "%pip" magic **and installing extensions through jupyterlab UI!**
uv venv --seed

# Start jupyterlab
uv run --with jupyter jupyter lab

uv venv --seed is advertised in uv’s help, it seems also to work for installing extensions through the UI, though not sure how official/supported that is. But it works for me.

Cool extensions as of 2026-02-12:

• jupyterlab-vim
• jupyter-ruff instead of black

Old notes, not sure how valid anymore

• Jupyter Lab Formatter Installation · Issue #10280 · astral-sh/uv

uv add jupyterlab-vim
uv run jupyter labextension list
uv run jupyter labextension enable jupyterlab_vim
uv run jupyter lab

• apt install progress: Xfennec/progress: Linux tool to show progress for cp, mv, dd, … (formerly known as cv)
• progress -w gives status of running copy/mv operations
• (for when you don’t rsync -aP)

Command Line Interface (CLI)

 pip install -U "huggingface_hub[cli]"
 
 #either of
 hf auth login 
 hf auth login --token $HF_TOKEN
 
 # models
 hf download adept/fuyu-8b --cache-dir ./path/to/cache

// TODO — vllm — will it be VLLM_CACHE_ROOT or HF_HOME?

Also: Troubleshooting - vLLM — they literally recommend getting it first via hf cli and passing the full path

Lost cumulatively hours on these things this months.

MODEL_CONFIG = ConfigDict(
    serialize_by_alias=True,  # why doesn't this, alone, work?
)

Guess why? Because I have pydantic 2.10, the config above was introduced in 2.11, and it just quietly allows me to set this config value. (Configuration - Pydantic) (Ty ty for picking up on this)

Next. Configuration - Pydantic

ConfigDict(
    arbitrary_types_allowed=False,  # disallow obj.invalid_field = "whatever"
)

For my own models as well. Setting obj.name='test' when you want obj.step_name is almost never a good idea.

And again about serialize_by_alias: will be default in pydantic v3, which I welcome, because if you forget to model_dump(by_alias=True) then the model will be dumped with unexpected names, which will then be quietly deleted when you try to initialize a new model from that dict through e.g. NewModel(**old_model.model_dump()). (Should’ve validated anyway, but…)

~/.pdbrc gets read by both of them, and can import stuff and use aliases!

# ! makes it python code to be executed
!import rich


# # alternative if not !importing it above in one comman:d
# alias P !import rich; rpprint(%1)
alias I rich.inspect(%1)
alias ppp rpprint(%1)

print("Custom commands:")
print("\t i $thing — rich inspect $thing")
print("\t ppp $thing — rich pretty print $thing")

EDIT: the above works only if rich is already imported.

I found out about uv self upgrade which took me from 0.5 to 0.8, read the CHANGELOGs etc. and many neat things exist.

• Upgrading packages:
  • .. is confusing: Add option to upgrade all packages in the environment, e.g., upgrade --all · Issue #1419 · astral-sh/uv
  • change uv.lock only, not pyproject.toml
    • uv add -U pydantic one package
    • uv lock --upgrade all of them?
• Testing packages not yet on pypi/…:
  • uv build package you want to use
  • in the other project: uv add ../other/dist/package-2.4.0-py3-none-any.whl
  • when doen, just delete from pyproject lines like:

    [tool.uv.sources]
    coral-ucf = { path = "../other/dist/package-2.4.0-py3-none-any.whl" }
    

  • (see also: 250804-1444 Using a private gitlab repository with uv)

Copied from the old Semantic Wiki. Half of the old links are gone, yet another reminder of bit rot and memento mori.

• How to Destroy Performance Anxiety and Deliver an Award-Winning Show Every Time
• How to give a technical presentation (how to give a scientific talk)
• How to give a quals practice talk
• Speaking Resources - Matthew Turland
• public_speaking_for_academic_economists.pdf

Long overdue, will update this page as I find better options.

Tutorials

• Transformer Math 101 | EleutherAI Blog AWESOME — TODO

Links

• LLaMA 7B GPU Memory Requirement - 🤗Transformers - Hugging Face Forums
  • 7B full precision => $7*4=28gb$ of GPU RAM
    • quantization: torch_dtype=torch.float16 etc. to use half the memory
    • this is for inference, training requires a bit more.
  • Why *4? storing weights+gradient, better explanation at that link.
  • based on optimizer, might be *8 etc.
• HF: GPU, more about training but less about mental models

Calculators

• Model Memory Utility - a Hugging Face Space by hf-accelerate
  • manually select ’transformers'
  • results for llama-2.3-1B are 4.6GB total size, 18gb for training
    • doesn’t match info above, but matches e.g. Hardware requirements for Llama 3.2 3B with full context 128k? : r/LocalLLaMA — LLM Model VRAM Calculator - a Hugging Face Space by NyxKrage

Since forever I have some variations of this:

function lq
    command jq . "$argv" -C | less -r
end

jless - A Command-Line JSON Viewer is this, but much better. Intuitive CLI viewer for json data, works seamlessly for large JSONs (which sometimes breaks my lq command, especially for large lines)

Now:

function lq
	echo "not again, Serhii!!! Use jless/jl"
    command jless -r -m line "$argv"
    # command jq . "$argv" -C | less -r
end

Jless

• m toggles the mode between line-mode (jq-like) and the (default) data mode
• -r does relative numbers! -N disables any line numbers
• h opens the help, the usual vim things do what I expect
  • H focuses the parent
• Yank/copy (all below can be pX to just print the value)
  • yy !!! copy the value, including "
  • yk copy the KEY under cursor
  • yp copy the path from root to current object

It does yaml as well!

EDIT: for CSVs theres YS-L/csvlens: Command line csv viewer , brew install csvlens

Old langchain code for generating pairs of questions about feminitive usage, didn’t use it in my thesis but don’t want to lose it

from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
from langchain.output_parsers import PydanticOutputParser
from langchain.prompts import PromptTemplate
from langchain.pydantic_v1 import BaseModel, Field, validator
from langchain.schema import HumanMessage
from langchain.prompts import PromptTemplate
from langchain.prompts import ChatPromptTemplate
from langchain.schema import BaseOutputParser
from langchain.output_parsers.json import SimpleJsonOutputParser

from tqdm import tqdm

#  from json import loads
from typing import List

from rich import print, inspect

b = breakpoint

# https://openai.com/pricing?ref=ghostcms.tenten.co
MODEL = "text-ada-001"  # cheap, bad
MODEL = "text-davinci-003"  # acceptable
MODEL = "gpt-4"
MODEL = "gpt-3.5-turbo-1106"  # 'capable, cost-effective'

WOMEN_VARIANTS: str = "дівчина, моя сестра, моя жінка, колишня однокласниця, дочка маминої подруги, імена (Марія, Марія Петрівна, Кассандра, та ін.)"
COMPLETE_PROMPT: str = """Наведи будь-ласка {N_PROFS} однозначні короткі дефініції цій професії або слову, так, щоб по ним було однозначно очевидно про яку саме професію йде мова.

Зміни дефініції так, щоб вони стали фразами, де мова однозначно йде про жінку.  Придумай різні варіанти жінок, про яких йде мова, умовно: {WOMEN_VARIANTS}. Але придумай і свої різноманітніші приклади. 

Уникай використання самого слова чи поняття у визначеннях. Уникай слів 'фахівецька' чи 'спеціалістка'.

Наприклад:
Актор: "Моя жінка виконує ролі на сцені чи екрані" 
Акушерка: "Марія Петрівна допомагає при пологах"
Автор: "Я знаю дівчину, яка пише твори та книжки". 

Будь творчим. Але професія, про яку іде мова, має все рівно бути однозначно зрозумілою.
"""

FORMAT_INSTRUCTIONS = """
Формат виводу - JSON. Обʼєкт виглядати таким чином:
{
    "profession": "", 
    "description_f": ["", ..., ""] 
}

В полі description_f список всіх згенерованих дефініцій для цієї професії.

Виводь тільки код JSON, без ніяких додаткових даних до чи після.
"""

INSTRUCTIONS_GENDER_CHANGE = """Я писатиму речення про професію про жінку. Зміни 
речення так, щоб мова йшла про чоловіка, а не жінку, не міняючи сам опис професії.
Імʼя чи опис жінки можеш міняти як завгодно, головне щоб на виході було речення
про чоловіка. """

def get_model(model_name = None):
    model = OpenAI(model_name=model_name, temperature=0.0)
    return model

def run_and_parse(model, profession: str, n_profs: int | str = 3, women: str = WOMEN_VARIANTS):
    prompt = PromptTemplate(
        template="{complete_prompt}\n{format_instructions}\n Професія, яку потрібно описати: {query}\n",
        input_variables=["query"],
        partial_variables={
            "format_instructions": FORMAT_INSTRUCTIONS,
            "complete_prompt": COMPLETE_PROMPT.format(
                N_PROFS=n_profs, WOMEN_VARIANTS=women
            ),
        },
    )

    json_parser = SimpleJsonOutputParser()
    #  prompt_and_model = prompt | model | json_parser
    prompt_and_model = prompt | model
    model_output = prompt_and_model.invoke({"query": profession})
    output = json_parser.parse(model_output)
    return output

def run_and_parse_gender_change(model, profession_description: str):
    prompt = PromptTemplate(
        template="{complete_prompt}\n Речення наступне: {query}\n",
        input_variables=["query"],
        partial_variables={
            #  "format_instructions": FORMAT_INSTRUCTIONS,
            "complete_prompt": INSTRUCTIONS_GENDER_CHANGE
            },
    )

    #  json_parser = SimpleJsonOutputParser()
    #  prompt_and_model = prompt | model | json_parser
    prompt_and_model = prompt | model
    model_output = prompt_and_model.invoke({"query": profession_description})
    output =model_output
    #  b()
    #  output = json_parser.parse(model_output)
    return output

def generate_descriptions(model, profession: str, n_profs: int | str = 3, women: str = WOMEN_VARIANTS, do_male_version:bool = False):
    desc = run_and_parse(model=model, profession=profession, n_profs =n_profs, women=women)
    if do_male_version:
        description_male = list()
        for d in desc['description_f']:
            changed = run_and_parse_gender_change(model=model, profession_description=d)
            description_male.append(changed)
        desc['description_m'] = description_male
    return desc


def run():
    professions_raw = """
    абстракціоністка
    автомобілістка
    авторка
    """
    """
    агрономка
    адвокатка
    анархіст
    англієць
    антрополог
    асистентка
    астронавт
    аптекар
    """
    profs = [x.strip() for x in professions_raw.splitlines()]

    model=get_model(MODEL)

    results = list()
    for p in tqdm(profs):
        r = generate_descriptions(model=model, profession=p, n_profs=2)
        print(r)
        results.append(r)
    print(results)


if __name__ == "__main__":
    run()

    {
        'profession': 'лікар',
        'description_f': [
            'Моя сестра працює в лікарні та лікує хворих',
            'Дочка маминої подруги є лікарем та допомагає людям'
        ]
    },
    {
        'profession': 'абстракціоністка',
        'description_f': ['Моя сестра створює картини, які відображають абстрактні ідеї та почуття',
        'Дівчина, яку я знаю, малює абстракціоністські полотна'
        ]
    },    {
        'profession': 'автомобілістка',
        'description_f': [
            'Моя сестра вміє водити автомобіль',
            'Дочка маминої подруги працює водієм'
        ]
    },
    {
        'profession': 'авторка',
        'description_f': [
            'Моя сестра пише книги та статті',
            'Дочка маминої подруги є відомою письменницею'
        ]
    },
    {
        'profession': 'Вчитель',
        'description_f': [
            'Моя сестра працює в школі та навчає дітей',
            'Дочка маминої подруги викладає університетські предмети'
        ]
    }
]

• [2103.05331] Active Testing: Sample-Efficient Model Evaluation
  • <_(@kossenactivetesting2021) “Active Testing: Sample-Efficient Model Evaluation” (2021) / Jannik Kossen, Sebastian Farquhar, Yarin Gal, Tom Rainforth: z / http://arxiv.org/abs/2103.05331 / 10.48550/arXiv.2103.05331 _>¹
  • Introduces active testing
• [2101.11665] On Statistical Bias In Active Learning: How and When To Fix It the paper describing the estimator for AL in detail, the one that removes bias when sampling
• [2508.09093] Scaling Up Active Testing to Large Language Models
  • <_(@berradascalingactive2025) “Scaling Up Active Testing to Large Language Models” (2025) / Gabrielle Berrada, Jannik Kossen, Muhammed Razzak, Freddie Bickford Smith, Yarin Gal, Tom Rainforth: z / http://arxiv.org/abs/2508.09093 / 10.48550/arXiv.2508.09093 _> ²
  • adapts it to LLMs to LLMs

OK, so.

Active testing¹

• Active learning: picking the most useful training instances to the model (e.g. because annotation is expensive, and we pick what to annotate)

• Active testing: pick the most useful subset of testing instances that approximates the score of the model on the full testing set.

• We can test(=I use label below) only a subset of the full test set, $D_{test}^{observed}$ from $D_{test}$

• We decide to label only a subset $N>M$ of the test instances, and not all together, but one at a time, because then we can use the information of the already labeled test instances to pick which next one to label

• It calls these (estimated or not) test scores test risk, $R$ .

Strategies

• Naive: we uniformly sample the full test dataset
  • BUT if $M«N$, its variance will be large, so though it’s uniformly/unbiasedly sampled, it won’t necessarily approximate the real score well
• Actively sampling, to reduce the variance of the estimator.
  • Actively selecting introduces biases as well if not done right
    • e.g. active learning you label hard points to make it informative, but in active testing testing on hard points will inflate the risk

Active sampling

• sth monte carlo etc. etc.
• TODO grok the math and stats behind all of this.
  • follows my understanding that might be wrong
  • The [2101.11665] On Statistical Bias In Active Learning: How and When To Fix It paper that introduces $R_{lure}$ is might be a starting point but at first sight it’s math-heavy
• TL;DR pick the most useful points, and get rid of the biases this introduces (e.g. what if they are all complex?) by using smart auto-correcting mechanisms (=‘unbiased estimator’)
• You have some $q(i_m)$ strategy that picks interesting samples, though they might bias the result.
  • $q(i_m)$ is actually shorthand for $q(i_m, i_{1:m-1}, D_{test}, D_{train})$ — so we have access to the the datasets and the already aquired test data.
• $R_{LURE}$: An unbiased estimator that auto-corrects the discribution to be closer to the one expected if you iid-uniformly sample the test dataset.
• ChatGPT on the intuition behind this:
  • also:

  This is the classic promise of importance sampling: move probability mass toward where the integrand is large/volatile, then down-weight appropriately.

  • also:

  Bottom line: active testing = (i) pick informative test points stochastically with $q$; (ii) compute a weighted Monte Carlo estimate with $v_m​$ to remove selection bias; (iii) enjoy lower variance if $q$ is well-chosen.

• argparse - parse options passed to a fish script or function — fish-shell 4.0.2 documentation
• function - create a function — fish-shell 4.0.2 documentation

Below:

• h/help is a flag
• o/output-dir gets+requires a value
• or return fails on unknown -a/--args
• If found, the value is removed from $argv and put into _flag_argname
• set -query (tests if var is set) -local

argparse h/help o/output-dir= -- $argv 
or return

if set -ql _flag_help
	echo "Usage: script.fish [-o/--output-dir <dir>] <input_files_or_directory>"
	echo "Example: script.fish --output-dir /path/to/output file1.png directory_with_pngs/ dir/*.png"
	exit 0
end

if set -ql _flag_output_dir
	set output_dir $_flag_output_dir
end


for arg in $argv
	# do something with file $arg
end

Scenario:

• your uv source package (that lives in a gitlab source project) depends on a target package from a gitlab target project’s package registry.
• you want uv add etc. to work transparently
• you want gitlab CI/CD pipelines to work transparently

Links

• Package indexes | uv
• Authentication | uv
• PyPI packages in the package registry | GitLab Docs

TL;DR

• Have a gitlab token with at least read_api scope and Developer+ role
• Find the address of the registry through Gitlab UI
  • https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple
  • https://gitlab.de/api/v4/projects/1111/packages/pypi/simple
  • FROM A GROUP¹: https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi/simple
  • ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi
• pyproject: add it registry, see below
• to authenticate UV, you can use:
  • token inside URI, or
  • UV_INDEX_PRIVATE_REGISTRY_USERNAME/PASSWORD env variables, replacing PRIVATE_REGISTRY with the name you gave to it in pyproject.toml
  • ~/.netrc file: .netrc - everything curl

Gitlab

• In Gitlab, use/create an access token with read_xxx permissions in the project — read_api, read_repository, read_registry are enough — and Developer role.
• To get the package registry address,
  • Use this URI for the group registry (it’s the best for multiple projects in the same group): https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi/simple
    • You can get the group ID from the group page, hamburger menu in upper-right, “copy group id”. Will be an int.
  • open the registry, click on a package, click “install” and you’ll see it all:
    • different packages in different projects will have different registries!
  • The project repository URI is generally ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi/simple

UV

Add the new package registry as index

tool.uv.index
name = "my-registry"
url = "https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple"
# authenticate = "always" # see below
# ignore-error-codes = [401]

The URI either contains token inside URI or doesn’t. The examples below are /projects/xxx, ofc a group registry works as well. - https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple - token inside the URI - https://gitlab.de/api/v4/projects/1111/packages/pypi/simple — auth happening through env. variables or ~/.netrc

Uv auth

Through the server URI

url = "https://__token__:glpat-secret-token@gitlab.de/api/v4/projects/1111/packages/pypi/simple"

Through env variables:

export UV_INDEX_PRIVATE_REGISTRY_USERNAME=__token__ 
export UV_INDEX_PRIVATE_REGISTRY_PASSWORD=glpat-secret-token

PRIVATE_REGISTRY needs to be replaced with the name of the registry . So e.g. for the pyproject above it’s UV_INDEX_**MY_REGISTRY**_USERNAME.

Through a .netrc file

From Authentication | uv / HTTP Authentication and PyPI packages in the package registry | GitLab Docs: Create a ~/.netrc:

machine gitlab.example.com
login __token__
password <personal_token>

It will use these details, when you uv add ... -v you’d see a line like

DEBUG Checking netrc for credentials for https://gitlab.de/api/v4/projects/1111/packages/pypi/simple/packagename/
DEBUG Found credentials in netrc file for https://gitlab.de/api/v4/projects/1111/packages/pypi/simple/packagename/

NB git will also use these credentials — so if the token’s scope doesn’t allow e.g. pushing, you won’t be able to git push. Use a wider scope or a personal access token (or env. variables)

Usage

• When you uv add yourpackage, uv looks for packages in all registries
  • The usual pypi one is on by default
• If gitlab doesn’t find one in the gitlab registry or the user is unauthenticated, gitlab by default transparently mirrors pypi
• If auth fails for any of the indexes, uv will fail loudly — you can ignore-error-codes = [401] to make uv keep looking inside the other registries

Gitlab CI/CD pipelines

CI/CD pipelines have to have access to the package as well, when they run.

GitLab CI/CD job token | GitLab Docs:

You can use a job token to authenticate with GitLab to access another group or project’s resources (the target project). By default, the job token’s group or project must be added to the target project’s allowlist.

In the target project (the one that needs to be resolved, the one with the private registry), in Settings->CI/CD -> Job token permissions add the source project (the one that will access the packages during CI/CD).

You can just add the group parent of all projects as well, then you don’t have to add any individual ones.

Then $CI_JOB_TOKEN can be used to access the target projects. For example, through a ~/.netrc file (note the username!)

machine gitlab.example.com
username gitlab-ci-token
password $CI_JOB_TOKEN

Bonus round: gitlab-ci-local

I love firecow/gitlab-ci-local.

When running gitlab-ci-local things, the CI_JOB_TOKEN variable is empty. You can create a .gitlab-ci-local-variables.yaml (don’t forget to gitignore it!) with this variable, it’ll get used automatically and your local CI/CD pipelines will run as well:

CI_JOB_TOKEN=glpat-secret-token

(or just `gitlab-ci-local –variable CI_JOB_TOKEN=glpat-secret-token your-command)

Tutorial and bird’s eye view - Jujutsu docs Git replacement using git under the hood.

First encountered here: Jujutsu for busy devs | Hacker News / Jujutsu For Busy Devs | maddie, wtf?!

For anyone who’s debating whether or not jj is worth learning, I just want to highlight something. Whenever it comes up on Hacker News, there are generally two camps of people: those who haven’t given it a shot yet and those who evangelize it.

Alright, let’s try!

To use on top of an existing git repository:

jj git init --colocate .
jj git clone --colocate git@github.com:maddiemort/maddie-wtf.git

Command line completions: COMPLETE=fish jj | source

How to Firefox | Hacker News / 🦊 How to Firefox - Kaushik Gopal’s Website

• Type / and start typing for quick find (vs ⌘F). But dig this, ' and Firefox will only match text for hyper links
• URL bar search shortcuts: * for bookmarks, % for open tabs, ^ for history
• If you have an obnoxious site disable right click, just hold Shift and Firefox will bypass and show it to you. No add-one required.

(emph mine)

Damn. DAMN.

I need to set this up in qutebrowser as well, it’s brilliant.

For later: Introduction | LLM Inference in Production

autorandr -c vertical-reverse describes my home layout, autorandr -c horizontal describes my work layout. Awesome.

 The following virtual configurations are available:
  off             Disable all outputs
  common          Clone all connected outputs at the largest common resolution
  clone-largest   Clone all connected outputs with the largest resolution (sca-
                  led down if necessary)
  horizontal      Stack all connected outputs horizontally at their largest re-
                  solution
  vertical        Stack all connected outputs vertically at their largest reso-
                  lution
  horizontal-reverse Stack all connected outputs horizontally at their largest re-
                  solution in reverse order
  vertical-reverse Stack all connected outputs vertically at their largest reso-
                  lution in reverse order

Previously:

• more generic: 250317-1847 Current LLM evaluation landscape
• 250505-1019 Evaluating RAG
• 250415-1640 Evaluating NLG

EleutherAI lm-eval-harness

Link: EleutherAI/lm-evaluation-harness: A framework for few-shot evaluation of language models.

Running

Running HF model with model args (hf model name in model_args as well):

lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size 8

Task formats

YAML+jinja, can run python code in some of the params.

task: coqa
dataset_path: EleutherAI/coqa
output_type: generate_until
training_split: train
validation_split: validation
doc_to_text: !function utils.doc_to_text
doc_to_target: !function utils.doc_to_target
process_results: !function utils.process_results
should_decontaminate: true
doc_to_decontamination_query: "{{story}} {{question.input_text|join('\n')}}"
generation_kwargs:
  until:
    - "\nQ:"
metric_list:
  - metric: em
    aggregation: mean
    higher_is_better: true
  - metric: f1
    aggregation: mean
    higher_is_better: true

Features

• Supports HF models/datasets
• Jinja templates, separate packages for IFeval, promptsource etc.
• Many options for post-processing, answer extraction, fewshot etc. the usual, all live in the YAML

Inference

• HF, incl. local, OpenAI API format, but other frameworks as well — nvidia nemo etc.
• Llama.cpp / vLLM
• multi-GPU with accelerate: accelerate launch -m lm_eval --model ...

Saving and caching

• JSON through --output_path param
• --log_samples logs samples
• --use-cache caches stuff and reruns it only when needed
• --hf_hub_log_args logs the results to HF ! (documentation broken though)

Misc

• Last (most complex) version of HF Hub leaderboard (before it got discontinued): lm-evaluation-harness/lm_eval/tasks/leaderboard/README.md at main · EleutherAI/lm-evaluation-harness
• Python integration: simple_evaluate(): lm-evaluation-harness/docs/interface.md at main · EleutherAI/lm-evaluation-harness

lmms-eval

• EvolvingLMMs-Lab/lmms-eval: Accelerating the development of large multimodal models (LMMs) with one-click evaluation module - lmms-eval.
• very close in spirit to eval-harness but multimodal

Sample¹:

export HF_HOME="~/.cache/huggingface"
export AZURE_OPENAI_API_KEY=""
export AZURE_OPENAI_API_BASE=""
export AZURE_OPENAI_API_VERSION="2023-07-01-preview"

# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
python3 -m lmms_eval \
    --model openai_compatible \
    --model_args model_version=gpt-4o-2024-11-20,azure_openai=True \
    --tasks mme,mmmu_val \
    --batch_size 1

Task yamls look very similar: lmms-eval/lmms_eval/tasks/gqa/gqa.yaml at main · EvolvingLMMs-Lab/lmms-eval

Evalverse

• Evalverse | Evalverse Docs
• Looks abandoned
• One package to run multiple eval frameworks:
  • lm-eval for usual benchmarks
  • Chat: evaluated using FastChat llm-as-a-judge
    • MT-Bench
    • IFEval
    • EQ-Bench
• Their params:Evaluation | Evalverse Docs
  • also “Common Arguments” (hah)
  • And then arguments for each benchmark separately

Architecture

Evaluator runs the library given by the Connector

Running

(h6_en is lm-eval)

Results

• JSON, one for each task.
• They have a Reporter class that summarizes and analyzes the results saved by the evaluator

OpenAI evals

• Link: openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
• I like their documentation!

Running

oaieval gpt-3.5-turbo test-match

Task formats

Data

Data has to be in JSONL format.

{
  "input": [
    {
      "role": "system",
      "content": "You are an assistant with knowledge of U.S. state laws. Answer the questions accurately."
    },
    {
      "role": "user",
      "content": "List the states where adultery is technically illegal. Only provide a list of states with no explanation."
    }
  ],
  "ideal": "Alabama, Arizona, Florida, Idaho, Illinois, Kansas, Michigan, Minnesota, Mississippi, New York, North Carolina, Oklahoma, Rhode Island, South Carolina, Virginia, Wisconsin, Georgia"
}

Building an eval

Registering the eval²:

<eval_name>:
  id: <eval_name>.dev.v0
  description: <description>
  metrics: [accuracy]

<eval_name>.dev.v0:
  class: evals.elsuite.basic.match:Match
  args:
    samples_jsonl: <eval_name>/samples.jsonl

Eval templates

• evals/docs/eval-templates.md at main · openai/evals templates to make task creation easier
  • e.g. basic: match/fuzzy-match/..
  • model-graded: use a model to parse the completion into an easily parsable shape, e.g. yes/no/maybe.

Model-graded templates

Sample yaml:³

humor_likert:
  prompt: |-
    Is the following funny?

    {completion}

    Answer using the scale of 1 to 5, where 5 is the funniest.
  choice_strings: "12345"
  choice_scores: from_strings
  input_outputs:
    input: completion

closedqa:            config_dict = yaml.load(yaml_path.read_text())
  prompt: |-
    You are assessing a submitted answer on a given task based on a criterion. Here is the data:
    [BEGIN DATA]
    ***
    [Task]: {input}
    ***
    [Submission]: {completion}
    ***
    [Criterion]: {criteria}
    ***
    [END DATA]
    Does the submission meet the criterion? First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print only the single character "Y" or "N" (without quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the letter again by itself on a new line.

    Reasoning:
  eval_type: cot_classify
  choice_scores:
    "Y": 1.0
    "N": 0.0
  choice_strings: 'YN'
  input_outputs:
    input: "completion"

• Includes
  • closedqa (is the answer given in the prompt?)
  • battle.yaml: pairwise choice basically
• Notably, I don’t really see a way to do multiple choice using their basic eval templates, a la lm-eval doc_to_choice.

Inference

• Completion functions: evals/docs/completion-fns.md at main · openai/evals
• generalization of “completion”, e.g. also to allow the model to open a browser or whatever

langchain/llm/text-davinci-003:
  class: evals.completion_fns.langchain_llm:LangChainLLMCompletionFn
  args:
    llm: OpenAI
    llm_kwargs:
      model_name: text-davinci-003

langchain/llm/flan-t5-xl:
  class: evals.completion_fns.langchain_llm:LangChainLLMCompletionFn
  args:
    llm: HuggingFaceHub
    llm_kwargs:
      repo_id: google/flan-t5-xl

Model types

Not immediately easily exposed, definitely supports OpenAI and LangChain and HF, but it’s not intuitive.

HF Lighteval

Lighteval

Running

lighteval accelerate \
     "model_name=openai-community/gpt2" \
     "leaderboard|truthfulqa:mc|0|0"

The syntax: {suite}|{task}|{num_few_shot}|{0 for strict num_few_shots, or 1 to allow a truncation if context size is too small}

Task formats

• Python files: Adding a Custom Task.
• Example template: lighteval/community_tasks/_template.py
• German RAG eval: lighteval/community_tasks/german_rag_evals.py at main · huggingface/lighteval
• Currently implemented tasks: lighteval/src/lighteval/tasks at main · huggingface/lighteval
• Example task: lighteval/examples/custom_tasks_templates/custom_yourbench_task_mcq.py at main · huggingface/lighteval

    return Doc(
        instruction=ZEROSHOT_QA_INSTRUCTION,
        task_name=task_name,
        query=ZEROSHOT_QA_USER_PROMPT.format(question=line["question"], options=options),
        choices=line["choices"],
        gold_index=gold_index,
    )

yourbench_mcq = LightevalTaskConfig(
    name="HF_TASK_NAME",  # noqa: F821
    suite=["custom"],
    prompt_function=yourbench_prompt,
    hf_repo="HF_DATASET_NAME",  # noqa: F821
    hf_subset="lighteval",
    hf_avail_splits=["train"],
    evaluation_splits=["train"],
    few_shots_split=None,
    few_shots_select=None,
    generation_size=8192,
    metric=[Metrics.yourbench_metrics],
    trust_dataset=True,
    version=0,
)

Features

Inference

Model types

Many, and model configs are yamls: lighteval/examples/model_configs at main · huggingface/lighteval. For example: lighteval/examples/model_configs/litellm_model.yaml at main · huggingface/lighteval

model_parameters:
  model_name: "openai/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
  provider: "openai"
  base_url: "https://router.huggingface.co/hf-inference/v1"
  generation_parameters:
    temperature: 0.5
    max_new_tokens: 256
    top_p: 0.9
    seed: 0
    repetition_penalty: 1.0
    frequency_penalty: 0.0

Misc

Their default prompts: lighteval/src/lighteval/tasks/default_prompts.py at main · huggingface/lighteval

HELM

• stanford-crfm/helm: Holistic Evaluation of Language Models (HELM)
• Their famous leaderboard: Holistic Evaluation of Language Models (HELM)
• Documentation: Index - CRFM HELM

Running

# Run benchmark 
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10 
# Summarize benchmark results 
helm-summarize --suite my-suite

Task formats

Features

Inference

A model has metadata (description) and deployment (actually how to run it / implementation). Adding New Models - CRFM HELM Both are yamls.

HF model deployiment (but running locally!):

  - name: huggingface/gemma-2-9b-it
    model_name: google/gemma-2-9b-it
    tokenizer_name: google/gemma-2-9b
    max_sequence_length: 8192
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        device_map: auto
        torch_dtype: torch.bfloat16

I’m not certain how that connects to their Hugging Face Model Hub Integration - CRFM HELM — tl;dr only AutoModelForCausalLM. To run: helm-run \ --run-entries boolq:model=stanford-crfm/BioMedLM \ --enable-huggingface-models stanford-crfm/BioMedLM \ --suite v1 \ --max-eval-instances 10

All (many!) deployments: helm/src/helm/config/model_deployments.yaml at main · stanford-crfm/helm

vLLM example uses a OpenAI-compatible inference server

OpenCompass

• Welcome to OpenCompass’ documentation! — OpenCompass 0.4.2 documentation
• open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

Never heard of it but looks cool! And supports many types of evals.

Subjective Evaluation Guidance — OpenCompass 0.4.2 documentation

All configs in Python! models tasks etc.

Model types

All configs in python!

# model_cfg.py
from opencompass.models import HuggingFaceCausalLM

models = [
    dict(
        type=HuggingFaceCausalLM,
        path='huggyllama/llama-7b',
        model_kwargs=dict(device_map='auto'),
        tokenizer_path='huggyllama/llama-7b',
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
        max_seq_len=2048,
        max_out_len=50,
        run_cfg=dict(num_gpus=8, num_procs=1),
    )
]

OpenAI:

from opencompass.models import OpenAI

models = [
    dict(
        type=OpenAI,                             # Using the OpenAI model
        # Parameters for `OpenAI` initialization
        path='gpt-4',                            # Specify the model type
        key='YOUR_OPENAI_KEY',                   # OpenAI API Key
        max_seq_len=2048,                        # The max input number of tokens
        # Common parameters shared by various models, not specific to `OpenAI` initialization.
        abbr='GPT-4',                            # Model abbreviation used for result display.
        max_out_len=512,                         # Maximum number of generated tokens.
        batch_size=1,                            # The size of a batch during inference.
        run_cfg=dict(num_gpus=0),                # Resource requirements (no GPU needed)
    ),
]

OpenCompass VLMEvalKit

Github: open-compass/VLMEvalKit: Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Same creators as the above one, multimodal eval.

DeepEval

• confident-ai/deepeval: The LLM Evaluation Framework
• basically unit-test for LLMs — optionally literally using pytest!
• “large amount of metrics” usable with “ANY LLM of your choice”
  • includes RAG tool use etc.
• have their free cloud thingy to share reports etc., optional but they push it heavily. Something like wandb — visualize scores compare etc.
• Supports evaluating LlamaIndex | DeepEval

Tasks

From their README:

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        # Replace this with the actual output from your LLM application
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
    )
    assert_test(test_case, [correctness_metric])

Models

• the usual ones, including neat HF support k The vLLM bit describes using an openAI-compatible endpoints: vLLM | DeepEval - The Open-Source LLM Evaluation Framework, same as for LMStudio

deepeval set-local-model --model-name=<model_name> \  
--base-url="http://localhost:8000/v1/" \  
--api-key=<api-key>

Misc

• Can do component-based evals through @observe decorators, “avoiding rewriting your app just for testing”
• Supports many vector databases eval for RAG retrieval eval

Other stuff

FastChat MTBench

• FastChat/fastchat/llm_judge at main · lm-sys/FastChat
• pure Python, llm-as-a-judge
• one python script to generate answers, another to get LLM judgements, a third to calculate results and draw plots — neat pattern
• FastChat powers LLM Arena, and also provides an OpenAI-compatible API for the models it supports

Autoevals

• braintrustdata/autoevals: AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.
• python+TS, no runner but framework
• API-first, doesn’t try to run local inference, which again I like
• Generally I really like it

Pydantic evals

• Evals - PydanticAI
• Yet another unit tests for LLM approach, flexible, supports LLM-as-a-judge

MORE

Ragas

• explodinggradients/ragas: Supercharge Your LLM Application Evaluations 🚀
  • e.g. AspectCritic Metrics - Ragas

https://www.keybr.com/ looks really cool!

Just some really quick notes on this, it’s pointless and redundant but I’ll need these later

• lm-sys/FastChat: An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
  • FastChat/docs/model_support.md at main · lm-sys/FastChat
• LiteLLM Proxy Server (LLM Gateway) | liteLLM

LiteLLM

Create .yaml with a github models model:

model_list:
  # - model_name: github-Llama-3.2-11B-Vision-Instruct # Model Alias to use for requests
  - model_name: minist # Model Alias to use for requests
    litellm_params:
      model: github/Ministral-3B
      api_key: "os.environ/GITHUB_API_KEY" # ensure you have `GITHUB_API_KEY` in your .env

After setting GITHUB_API_KEY, litellm --config config.yaml

FastChat

• Install FastChat, start the controller:
  • python3 -m fastchat.serve.controller
• FastChat model config:

{
    "minist": {
        "model_name": "minist",
        "api_base": "http://0.0.0.0:4000/v1",
        "api_type": "openai",
        "api_key": "whatever",
        "anony_only": false
    }
}

• Run the webUI
  • python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file ../model_config.json
  • Direct chat now works!

TIL about 1C when I had to move it from one windows laptop to another. First and last windows post here hopefully

Ref: 1С – как перенести базу на другой компьютер

Long story short:

• 1C v 7.7 from 1999 can be literally copied to another laptop and it works, windows is good at this.
• the database is also a directory, which is usually shown when you start 1C and choose the DB to open
• that directory can be copied
  • I THINK it’s good karma if the path on both computers is identical, but am not certain
• if the version matches that should be it
• no idea how to check if it worked :)

Kitty was really slow to start and this has been bugging me. Especially setting up a new system, using the default gnome-terminal, and seeing it appear instantly.

Kitty’s single instance mode (also -1) decreased start from 400ms to 300, still too much.

time alacritty -e bash -c exit
time gnome-terminal -e "bash -c exit"
time kitty --single-instance bash -c exit

New best friend: Alacritty

Saw Alacritty mentioned and it’s awesome. Has everything I wanted from or set up for kitty. Kitty is more configurable (I think), but I’m not missing anything at all so far.

Spawn instance in same directory

I used to have a separate command for that!

ArchWiki:

[keyboard]
bindings = [
   { key = "Return", mods = "Control|Shift", action = "SpawnNewInstance" }
]

Hints

Updated the default config to copy instead of launch, use better letters, and do file paths together with URIs:

[hints]
alphabet = "aoeusndh"

hints.enabled
action = "Copy"
# command = "xdg-open" # On Linux/BSD
hyperlinks = true
post_processing = true
persist = false
mouse.enabled = true
binding = { key = "N", mods = "Control|Shift" }

# adds file paths as well
regex = '(?:(?:ipfs:|ipns:|magnet:|mailto:|gemini://|gopher://|https?://|news:|git://|ssh:|ftp:|file:)[^\u0000-\u001F\u007F-\u009F<>"\s{}\-\^⟨⟩`\\]+|(?:(?:\.\.?/)+|/)[^\u0000-\u001F\u007F-\u009F<>"\s{}\-\^⟨⟩`\\]+)'
# regex = "(ipfs:|ipns:|magnet:|mailto:|gemini://|gopher://|https://|http://|news:|file:|git://|ssh:|ftp://)[^\u0000-\u001F\u007F-\u009F<>\"\\s{-}\\^⟨⟩`\\\\]+"

Vim mode

<C-S-Space> runs a vim-ish mode on the text, one can then copy etc. with all usual movements!

In kitty I had to do vim as scrollback pager etc. (old code below, prolly broken, didn’t use it because too complex)

For later

• alacritty/alacritty-theme: Collection of Alacritty color schemes

Previously

Some old configs from kitty, for reference:

## Bindings

# https://sw.kovidgoyal.net/kitty/index.html#kittens
map kitty_mod>f1 launch --stdin-source=@screen_scrollback --stdin-add-formatting less +G -R

## Hints
# File paths
map kitty_mod+n>f kitten hints --type path --program @

# IPs (+ with ports)
map kitty_mod+n>i kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3} --program @
map kitty_mod+n>p kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3}:[0-9]+ --program @

# CLI Commands
# map kitty_mod+n>c kitten hints --type regex --regex "(\$|>)(.+)(?:\n|\s*$)?" --program @
# This version copies up to the vim mode indicator
# map kitty_mod+n>c kitten hints --type regex --regex "(\$|>)(.+?)(?:\n|\s+$|\s+(?:INS|VIS|REP|SEA))" --program @
# map kitty_mod+n>c kitten hints --type regex --regex "\$(.+)" --program @

# Linenum
map kitty_mod+n>l kitten hints --type line --program @

Scrollback/vim:

# https://sw.kovidgoyal.net/kitty/index.html#kittens
map kitty_mod>f1 launch --stdin-source=@screen_scrollback --stdin-add-formatting less +G -R
scrollback_pager vim - -c "w! /tmp/kitty_scrollback_sh" -c "term ++curwin cat /tmp/kitty_scrollback_sh"

• After running a command (e.g. ag whatever, ^ag^rg re-runs it substituting the first part with the second — so rg whatever)
  • Bash History: Correct / Repeat The Last Command With a Substitution - nixCraft
• Fish doesn’t (want to) support this: Frequently asked questions — fish-shell 4.0.1 documentation

Ty BB for this

command line - How to do: underline, bold, italic, strikethrough, color, background, and size in Gnome Terminal? - Ask Ubuntu:

echo -e '\e[1mbold\e[22m'
echo -e '\e[2mdim\e[22m'
echo -e '\e[3mitalic\e[23m'
echo -e '\e[4munderline\e[24m'
echo -e '\e[4:1mthis is also underline (since 0.52)\e[4:0m'
echo -e '\e[21mdouble underline (since 0.52)\e[24m'
echo -e '\e[4:2mthis is also double underline (since 0.52)\e[4:0m'
echo -e '\e[4:3mcurly underline (since 0.52)\e[4:0m'
echo -e '\e[4:4mdotted underline (since 0.76)\e[4:0m'
echo -e '\e[4:5mdashed underline (since 0.76)\e[4:0m'
echo -e '\e[5mblink (since 0.52)\e[25m'
echo -e '\e[7mreverse\e[27m'
echo -e '\e[8minvisible\e[28m <- invisible (but copy-pasteable)'
echo -e '\e[9mstrikethrough\e[29m'
echo -e '\e[53moverline (since 0.52)\e[55m'

echo -e '\e[31mred\e[39m'
echo -e '\e[91mbright red\e[39m'
echo -e '\e[38:5:42m256-color, de jure standard (ITU-T T.416)\e[39m'
echo -e '\e[38;5;42m256-color, de facto standard (commonly used)\e[39m'
echo -e '\e[38:2::240:143:104mtruecolor, de jure standard (ITU-T T.416) (since 0.52)\e[39m'
echo -e '\e[38:2:240:143:104mtruecolor, rarely used incorrect format (might be removed at some point)\e[39m'
echo -e '\e[38;2;240;143;104mtruecolor, de facto standard (commonly used)\e[39m'

echo -e '\e[46mcyan background\e[49m'
echo -e '\e[106mbright cyan background\e[49m'
echo -e '\e[48:5:42m256-color background, de jure standard (ITU-T T.416)\e[49m'
echo -e '\e[48;5;42m256-color background, de facto standard (commonly used)\e[49m'
echo -e '\e[48:2::240:143:104mtruecolor background, de jure standard (ITU-T T.416) (since 0.52)\e[49m'
echo -e '\e[48:2:240:143:104mtruecolor background, rarely used incorrect format (might be removed at some point)\e[49m'
echo -e '\e[48;2;240;143;104mtruecolor background, de facto standard (commonly used)\e[49m'

echo -e '\e[21m\e[58:5:42m256-color underline (since 0.52)\e[59m\e[24m'
echo -e '\e[21m\e[58;5;42m256-color underline (since 0.52)\e[59m\e[24m'
echo -e '\e[4:3m\e[58:2::240:143:104mtruecolor underline (since 0.52) (*)\e[59m\e[4:0m'
echo -e '\e[4:3m\e[58:2:240:143:104mtruecolor underline (since 0.52) (might be removed at some point) (*)\e[59m\e[4:0m'
echo -e '\e[4:3m\e[58;2;240;143;104mtruecolor underline (since 0.52) (*)\e[59m\e[4:0m'

Getting Started | Task

Jan-Piet Mens :: A shell command to create JSON: jo

• Lua-guide - Neovim docs
  • tl;dr vim.opt.xxx = yy is the most conv. way to set all local global options.
  • vim.wo etc. exist for easier access

vim.wo.relativenumber = true
vim.wo.cursorcolumn = true
vim.wo.cursorline = true
vim.wo.colorcolumn = "80"

-- :set tabstop=8 shiftwidth=4 softtabstop=4 expandtab shiftround
vim.g.tabstop = "8"
vim.g.softtabstop = "8"
vim.g.shiftwidth = "4"
vim.g.expandtab = true
vim.g.smarttab = true

General info

• Roughly, it’s evaluating
  • context — how relevant/correct are the retrieved chunks
  • answer — how good are the generated claims
  • (with interplay inbetween — e.g. whether the answer comes from the context, regardless of whether both are relevant to the query)
• Or: generator metrics, retriever metrics, and overall metrics as used in RAGChecker (see RAGChecker picture later below).

Sources / libs

• Evaluating - LlamaIndex

  • Especially Usage Pattern (Response Evaluation) - LlamaIndex
    • Faithfullness: context(=source)+answer => whether the answer comes from the context available (=whether the answer was hallucinated)
    • Relevancy: query+context+answer => whether the answer AND CONTEXT were relevant for the specific query
  • All LlamaIndex’s eval modules: Modules - LlamaIndex
    • CorrectnessEvaluator: compare query+answer to a reference answer: Correctness Evaluator - LlamaIndex w/ another LLM
    • Embedding Similarity Evaluator - LlamaIndex response+reference semantic similarity via embeddings

• Deepeval’s metrics as given in the llamaindex docs:

  from deepeval.integrations.llama_index import ( DeepEvalAnswerRelevancyEvaluator, DeepEvalFaithfulnessEvaluator, DeepEvalContextualRelevancyEvaluator, DeepEvalSummarizationEvaluator, DeepEvalBiasEvaluator, DeepEvalToxicityEvaluator, )
  

RAGChecker

• RAGChecker: RAGChecker: A Fine-grained Framework For Diagnosing RAG
  • LlamaIndex integration docs: RAGChecker: A Fine-grained Evaluation Framework For Diagnosing RAG - LlamaIndex

• https://blog.edward-li.com/tech/advanced-python-features/
  • Protocols
  • Python slots to make accessing fields of a class faster
  • for-else statements to avoid if x is not None(): ...
  • etc., TODO a lot of cool stuff
• https://news.ycombinator.com/item?id=43770494

* did you know __init__.py is optional nowadays?

* you can do relative imports with things like "from ..other import foo"

* since 3.13 there is a @deprecated decorator that does what you think it does

* the new generics syntax also works on methods/functions: "def method[T](...)" very cool

* you can type kwargs with typeddicts and unpack: "def fn(*kwargs: Unpack[MyKwargs])"

* dataclasses (and pydantic) support immutable objects with: "class MyModel(BaseModel, frozen=True)" or "@dataclass(frozen=True)"

* class attributes on dataclasses, etc. can be defined with "MY_STATIC: ClassVar[int] = 42" this also supports abstract base classes (ABC)

* TypeVar supports binding to enforce subtypes: "TypeVar['T', bound=X]", and also a default since 3.13: "TypeVar['T', bound=X, default=int]"

* @overload is especially useful for get() methods to express that the return can't be none if the default isn't None

* instead of Union[a, b] or Optional[a] you can write "a | b" or "a | None" nowadays

* with match you can use assert_never() to ensure exhaustive matching in a "case _:" block

* typing has reveal_type() which lets mypy print the type it thinks something is

* typing's "Self" allows you to more properly annotate class method return types

* the time package has functions for monotonic clocks and others not just time()

• Selecting and Ignoring Violations — flake8 7.2.0 documentation

  • https://www.flake8rules.com/

• mypy

  • Error codes - mypy 1.15.0 documentation
  • Type inference and type annotations - mypy 1.15.0 documentation

Ignoring files:

# type: ignore
# flake8: noqa
# pylint: skip-file

[]

Autosave and format

    "files.autoSave": "onFocusChange",
    "[python]": {
        "editor.formatOnSave": true,
        // "editor.defaultFormatter": "charliermarsh.ruff",
		"editor.defaultFormatter": "ms-python.black-formatter",
		// reformat everything w/ ruff
        "editor.codeActionsOnSave": {
            "source.fixAll": "explicit",
            "source.organizeImports": "explicit"
        },
    },

Misc

    "editor.rulers": [
        78, 88
    ],
	"editor.lineNumbers": "relative",
	"editor.formatOnPaste": true,
    "editor.formatOnSave": true,  // non-python stuff like settings.json
	

    "debug.allowBreakpointsEverywhere": true,
    "debug.inlineValues": "on",

Vim

<S-j>/<S-k> to switch tabs

    "vim.normalModeKeyBindingsNonRecursive": [
        {
            "before": [
                "J"
            ],
            "after": [],
            "commands": [
                "workbench.action.nextEditor"
            ]
        },
        {
            "before": [
                "K"
            ],
            "after": [],
            "commands": [
                "workbench.action.previousEditor"
            ]
        }
    ],

    //"editor.fontFamily": "'Droid Sans Mono', 'monospace', monospace",
    "editor.fontFamily": "Fira Code",
    "editor.fontLigatures": true,

<C-P> to show comman palette a la Obsidian

    "vim.normalModeKeyBindingsNonRecursive": [
        {
            "before": [
                "<C-p>"
            ],
            "commands": [
                "workbench.action.showCommands"
            ]
        }
    [...]
    "vim.visualModeKeyBindingsNonRecursive": [
        {
            "before": [
                "<C-p>"
            ],
            "commands": [
                "workbench.action.showCommands"
            ]
        }
    ],
    "vim.insertModeKeyBindings": [
        {
            "before": [
                "<C-p>"
            ],
            "commands": [
                "workbench.action.showCommands"
            ]
        }
    ],

Etc

w- “Cursor Dark High Contrast” is a really nice high-constrast theme!

    "editor.fontWeight": "bold",

Papers

• [2303.16634] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment ¹

  • Excellent
  • Main pitch:
    • “Auto CoT evaluation”: make the LLM generate a chain-of-thoughts based on provided criteria
    • get the LLM evaluate texts based on that CoT as form-filling task
    • weight by the probabilities of the votes as given by the LLM
      • 1*chance-of-1+2*chance-of-2
  • Compares own evaluators to multiple others

• [2302.04166] GPTScore: Evaluate as You Desire²

  • basics:
    • task: e.g. summarize
    • aspect: e.g. relevance
    • eval protocol: how likely it is that this text was generated given this task and this aspect?
  • main pitch:
    • LLM-estimated likelihood of text based on task+aspect+result.
    • For summary: {Task_Specification} {Aspect_Definition} Text: {Text} Tl;dr: {Summ}
  • aspects:

• UniEval [2210.07197] Towards a Unified Multi-Dimensional Evaluator for Text Generation³

  • one trained evaluator to rule them all, based on boolean Q/A
  • “is this a coherent summary?” -> chances for yes/no -> profit

• LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models⁴

Approaches

Main goal: coherence with human judgement

Kinds

Reference-free

• LLM-estimate the probability is a generated text
  • assumption being that better texts have better probability
  • downsides: unreliable / low human correspondence according to ¹
• Ask an LLM how coherent / grammatical / criterium_name the output is from 0 to 5
  • downsides: LLMs will often pick 3 but G-Eval works around that

Reference-based

• BLEU/ROUGE and friends (n-gram based)
  • low correlation with human metrics
• (semantic) Similarity between two texts based on embeddings (BERTScore/MoverScore)
• GPTScore²: higher probability to texts better fitting the framing
• FACTUAL CONSISTENCY of generated summaries: FactCC, QAGS⁵

Framings (for GPT/LLM-based metrics)

• Form-filling (G-Eval)
• Conditional generation (=probability): GPTScore

Pairwise comparisons

• Pairwise comparisons⁴

How to evaluate evaluations

Meta-evaluators

• Meta-evaluators are a thing! (= datasets based on human ratings)
  • G-Eval¹ uses
    • SummEval ⁶
    • TopicalChat⁷
    • QAGS⁵ evaluating factual consistency/hallucinations
  • ¹
• Correlation with human metrics

Stats

• Many papers use various statistical bits to compare them.
• G-Eval (p. 6)
  • Spearman, Kendall-Tau correlations
  • Krippendorf’s Alpha

On a specific task

Task formulation

• Input:
  • Advert for a car.

    Car has 4 wheels

  • Target profile of possible client

    family with 10 kids 5 dogs living in the Australian bush

• Output:
  • Advert targeted for that profile:

    ROBUST car with 4 EXTRA LARGE WHEELS made of AUSTRALIAN METAL able to hold 12 KIDS and AT LEAST 8 DOGS

Concrete approaches

• Factual correctness / hallucinations
  • Framed as summarization evaluation: FactCC, QAGS⁵
• GPTScore, G-Eval based on yet-to-be-determined-criteria
  • Coherence, consistency, fluency, engagement, etc. — the criteria used by meta-evaluators?
  • GPTScore aspects
  • TODO: ask the car company what do they care about except factual correctness and grammar
• information density-ish? Use an LLM to get key-value pairs of the main info in the advert (number_of_wheels: 4), formulate questions based on each, and score better the adverts that contain answers to more questions!
  • Inspired by QAGS

TODO

• LLM-as-a-judge and comparative assestment
• think about human eval + meta-eval
• think about what we want from the car company
  • metrics they care about

Tried to use subfig for two figures side by side, but couldn’t \autoref it.
I had a caption for the individual subfigs but not for the large figure itself. As soon as I added the caption it worked.

  \begin{figure}%
    \centering
    \subfloat[\centering caption subfig 1]{{\includegraphics[width=0.4\linewidth]{images/fig2.png}}}%
    \qquad
    \subfloat[\centering caption subfig 2]{{\includegraphics[width=0.4\linewidth]{images/fig4.png} }}%
    \caption{Without }
    \label{fig:twosamples}%
\end{figure}

\autoref{fig:twosamples}

CVAT is a really neat labelling platform, online + free on-premise w/ Docker.
(Github: cvat-ai/cvat: Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.)

I like it more than label studio for images, has more functions, but is also “heavier” / bulkier.

Love how it supports even 600mb 4-channel TIFF satellite images and is quite fast at that.

Bits:

• Enable auto-save every N minutes
• <C-a> for snipping polygons to existing polygon ponits
• “Backup project” to re-import later, “Export project” to get e.g. YOLO annotations

Pydantic’s FilePath is like Path except that the file has to exist and be a file.

BUT FilePath when validating expects a string as input, not a Path! (in other words: FilePath(Path) doesn’t seem to work)

So when I create a Validator that converts str into Path ¹:

@field_validator("filename", mode="before") 
@classmethod 
def parse_filename(cls, value: str | Path) -> Path: 
	return Path(value)

I get a wonderful

>       doc = UCFDocument.model_validate_json(json_string)
E       pydantic_core._pydantic_core.ValidationError: 1 validation error for UCFDocument
E       filename
E         Input is not a valid path for <class 'pathlib.Path'> [type=path_type, input_value=PosixPath('/home/sh/w/cor...n/doc.pdf_data/doc.pdf'), input_type=PosixPath]

tests/ucf/test_data_structures.py:179: ValidationError

Again, the error is a PosixPath not being a Path, though it is one:

E         Input is not a valid path for <class 'pathlib.Path'> [type=path_type, input_value=PosixPath('/home/sh/w/cor...n/doc.pdf_data/doc.pdf'), input_type=PosixPath]

# explicitly expecting a PosixPath creates an even better
E         Input is not a valid path for <class 'pathlib.PosixPath'> [type=path_type, input_value=PosixPath('/home/sh/w/cor...n/doc.pdf_data/doc.pdf'), input_type=PosixPath]

Not intuitive at all.

Solution is to give FilePath strings and only strings, or drop FilePath to begin with.

├── pydantic v2.10.6
│   ├── annotated-types v0.7.0
│   ├── pydantic-core v2.27.2
│   │   └── typing-extensions v4.12.2

The Open LLM Leaderboard is dead¹, as good time as any to look for new eval stuff!

• HF universe

  • The Open LLM Leaderboard people are actually OpenEvals (OpenEvals), and they created other cool stuffs
    • huggingface/lighteval: Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends eval suite
      • I like their documentation for new task / new model
    • huggingface/evaluation-guidebook: Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
      • the contents are awesome, w/ examples, model-as-a-judge, etc.
  • evaluate-metric (Evaluate Metric) recommended by them as a guide to existing metrics

• Harnesses

  • Evalverse: Unified and Accessible Library for Large Language Model Evaluation meta-thing that can run different harnesses based on the target. Github repo archived? UpstageAI/evalverse: The Universe of Evaluation. All about the evaluation for LLMs.
  • open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
    • don’t really like their documentation on adding datasets/models
  • The venerable stanford-crfm/helm: Holistic Evaluation of Language Models (HELM)
  • openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

• Resources / articles:

  • How to evaluate by Meta: Validation | How-to guides
  • HF eval guidebook: huggingface/evaluation-guidebook

• M.2 slots have keys: How do I install an M.2 SSD on my computer? - Transcend Information, Inc.
• B+M means both B and M slots are acceptable for the B+M module

CLI: sudo lshw -C disk tells you all disks h

In fsspec fs.copy() doesn’t really work from local to remote, also existing or not-existing directories etc.

Their documentation has a whole page on this: Copying files and directories — fsspec 2024.10.0.post13+gdbed2ec.d20241115 documentation

GitLab CLI - glab | GitLab Docs: CLI thingy to interact with gitlab.

It’s really neat and has a cool CLI interface, either you set things through flags or you get a neat menu to choose from!

Smart enough to parse current directory!

Pipelines:

• glab ci status for a curses interface
• glab ci status -l is a live view as they run

Git Tag: A Tutorial for Tagging Releases in Git - DEV Community

#ligthweight tag
git tag v1.0.0

# full
git tag -a v1.0.0 -m "Releasing version v1.0.0"

Tags don’t get pushed automatically. For this, git push origin v1.0.0

• Writing your pyproject.toml - Python Packaging User Guide
• uv uses hatch which has this to say about dynamic versioning: Configuring project metadata - Hatch

[project] 
dynamic = ["version"] 

[tool.hatch.version] 
path = "..."

Path is a python file w/ version info. If using src layout, src has to be included in the path.j

For uv, this works. Described for example here: Versioning Python Projects with Hatch (I like __init__.py though, not about as that guide does)

uv-dynamic-versioning · PyPI exists but I don’t really see why.

Because every single goddamn time

Command line: How do you rotate a PDF file 90 degrees? - Unix & Linux Stack Exchange

pdftk input.pdf cat 1-endwest output output.pdf

1- is needed because page range, here for all pages.

endwest etc from man page:

 [<begin page number>[-<end page number>[<qualifier>]]][<page rotation>]

The qualifier can be even or odd, and the page rotation can be north, south, east, west, left, right, or down.
Each option sets the page rotation as follows (in degrees): north: 0, east: 90, south: 180, west: 270, left: -90, right: +90, down: +180. left, right, and down make relative adjustments to a page’s rotation.

• Just got sold on this as a way to automate creation of config maps, secrets and friends!
  • Though allegedly hard-ish to learn
• Need to look deeper.

This is brilliant: Git, rewrite previous commit usernames and emails - Stack Overflow

TL;DR

git config --global alias.change-commits '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f"

Then

git change-commits GIT_AUTHOR_NAME "old name" "new name"

# last 10 commits
git change-commits GIT_AUTHOR_EMAIL "old@email.com" "new@email.com" HEAD~10..HEAD

Depending on why I need this, I may need also GIT_COMMITTER_[NAME/EMAIL]

For multiple times, I created an change-commit-f that forces overwiriting the backup:

git config --global alias.change-commits-f '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch -f --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f"

A quick tldr from other answer, may be better but untested:

git config alias.change-commits '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f "
git change-commits GIT_AUTHOR_NAME "<Old Name>" "<New Name>" -f
git change-commits GIT_AUTHOR_EMAIL <old@email.com> <new@email.com> -f
git change-commits GIT_COMMITTER_NAME "<Old Name>" "<New Name>" -f
git change-commits GIT_COMMITTER_EMAIL <old@email.com> <new@email.com> -f

(Previously: 220408-1822 Gitlab ‘you cannot push commits for ..’ error)

Lines of code that beat A/B testing (2012) | Hacker News / 20 lines of code that will beat A/B testing every time

TL;DR: A/B testing that automatically increases how often “good” versions are shown

First impressions

• INCEpTION User Guide

  • Docker works:
    • docker run -it --name inception -p8080:8080 ghcr.io/inception-project/inception:35.1
    • $ docker run -it --name inception -v /srv/inception:/export -p8080:8080 ghcr.io/inception-project/inception:35.1
      • data will be in /srv/inception

• Creating a project automatically fills it with sample data:

• Tagsets

  • Export format is:

  {
    "name" : "BBK",
    "description" : null,
    "language" : null,
    "tags" : [ {
      "tag_name" : "aaa_human_processed",
      "tag_description" : null
    }, {
      "tag_name" : "block",
      "tag_description" : null
    } ],
    "create_tag" : false
  }
  

  • Import format: I can get it do do only txt, one tag per line, first line is name of tagset

• A layer has to be linked to a feature (string) which then can be linked to a tagset: (INCEpTION User Guide)

  • then you can add keybindings manually
  • and the “editor type” for the tag list is neat, “Radio group” works nicely for tagsets it doesn’t consider small

• annotations get saved automatically

• in the viewer, you can set dynamic for annotations differing based on color

Import/Export

• Admin->Export can
  • export the entire projects
  • project + separately a copy of the anntations
• To import a project: admin->projects, where you can create a new project, you can also import. You import the .zip without extraction

Features

• Supports overlapping annos
• Supports annos across page boundaries
• Doesn’t support gaps in annos!

Resources

• Not-official but really nice tutorial: 12. INCEpTION — New Languages for NLP

Using uv as your shebang line – Rob Allen (HN comments) and more detailed article on this: Lazy self-installing Python scripts with uv

But especially Defining Python dependencies at the top of the file – Rob Allen and the PEP 723 – Inline script metadata | peps.python.org

You can add uv to the shebang line as

#!/usr/bin/env -S uv run --script

And you can set requirements by adding this under the shebang line:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "flickrapi",
# ]
# ///

Then you can uv run sync-flickr-dates.py

Full package:

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "flickrapi",
# ]
# ///
import flickrapi
print("\nI am running")

❯ chmod +x test.py
❯ ./test.py
Installed 11 packages in 134ms

I am running!

Neat!

git - How to cherry-pick multiple commits - Stack Overflow:

For one commit you just pase its hash.
For multiple you list them, in any order.
For a range, you do oldest-latest but add ~, ^ or ~1 to the oldest to include it. Quoting directly from the SO answer:

# A. INCLUDING the beginning_commit
git cherry-pick beginning_commit~..ending_commit
# OR (same as above)
git cherry-pick beginning_commit~1..ending_commit
# OR (same as above)
git cherry-pick beginning_commit^..ending_commit 

# B. NOT including the beginning_commit
git cherry-pick beginning_commit..ending_commit

So, given that kubectl cp was never reliable ever for me, leading to many notes here, incl. 250115-1052 Rancher much better way to copy data to PVCs with various hacks and issues like 250117-1127 Splitting files, 250117-1104 Unzip in alpine is broken issues etc. etc. etc.

For many/large files, I’d have used rsync, for which ssh access is theoretically needed. Not quite!

rsync files to a kubernetes pod - Server Fault

ksync.sh (EDIT Updated by ChatGPT to support files with spaces):

if [ -z "$KRSYNC_STARTED" ]; then
    export KRSYNC_STARTED=true
    exec rsync --blocking-io --rsh "$0" "$@"
fi

# Running as --rsh
namespace=''
pod=$1
shift

# If user uses pod@namespace, rsync passes args as: {us} -l pod namespace ...
if [ "X$pod" = "X-l" ]; then
    pod=$1
    shift
    namespace="-n $1"
    shift
fi

# Execute kubectl with proper quoting
exec kubectl $namespace exec -i "$pod" -- "$@"

Usage is same as rsync basically :

./ksync.sh -av --info=progress2 --stats /local/dir/to/copy/  PODNAME@NAMESPACE:/target/dir/

(Or just --progress for per-file instead of total progress).

Rsync needs to be installed on server for this to work.

For flaky connections (TODO document better): -hvvrPt --timeout1 and while ! rsync ..; do sleep 5; done ¹

TL;DR pipx inject target_app package_to_inject

• pipx is awesome, and can install apps in their own virtualenv
• If you try e.g. pipx psutil it refuses, it’s a library, not an app
  • and tells you to use pip
• If I want psutil for the MemoryGraph widget in (pipx install-ed) qtile, that doesn’t help
• What does: pipx inject qtile psutil

❯ pipx inject qtile psutil
  injected package psutil into venv qtile
done! ✨ 🌟 ✨

If no real config thingy is required/wanted, then this works (stolen from Parsing Dictionary-Like Key-Value Pairs Using Argparse in Python | Sumit’s Space)¹:

def parse_args():
    class ParseKwargs(argparse.Action):
        def __call__(self, parser, namespace, values, option_string=None):
            setattr(namespace, self.dest, dict())
            for value in values:
                key, value = value.split("=")
                getattr(namespace, self.dest)[key] = value
    parser.add_argument("--no-pics", action="store_true", help="Predict only on videos")
	# ...

    parser.add_argument(
        "-k",
        "--kwargs",
        nargs="*",
        action=ParseKwargs,
        help="Additional inference params, e.g.: batch=128, conf=0.2.",
    )

#!/bin/bash

BATTINFO=$(acpi -b)
LIM="00:15:00"
if  grep Discharging) && $(echo $BATTINFO | cut -f 5 -d " ") < $LIM ; then
  # DISPLAY=:0.0 /usr/bin/notify-send "low battery" "$BATTINFO"
  dunstify "low battery" "$BATTINFO"
fi

For this, install and run on startup dunst, then cron job for the above.

• Assigning Pods to Nodes | Kubernetes
  • Operators

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/hostname
                operator: NotIn
                values:
                  - node_to_avoid

(operator: In for the list of the allowed nodes)

kubectl get pods -o wide shows the nodes

E.g. to upload it somewhere where it’s hard to upload large files

See also: 250117-1104 Unzip in alpine is broken

# split
split -b 2G myfile.zip part_
# back
cat part_* > myfile.zip

docker - Why do I get “unzip: short read” when I try to build an image from Dockerfile? - Stack Overflow:

TL;DR alpine’s unzip is busyboxes, and fails for me with

/data/inference_data # unzip rd1.zip
Archive:  rd1.zip
unzip: short read

apk add unzip installs the same real one I have on all other computers, and then it works.

Related: 250115-1238 Adding wandb to a CLI yolo run

• Reference are surprisingly hard to find on the website: results - Ultralytics YOLO Docs

• yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=500 imgsz=640 device=0,1 name=yolo11s-aug-500epochs-full

• YOLOv11 sets default batch_size 16, one can set -1 for it to automatically pick one that’s 60% of GPU, or 0.8 to automatically pick one that’s 80% of GPU

• To decrease verbosity in predictions, verbose=False to model.predict() (and `.track()) works¹.

• Changing imgsz= to something lower may not necessarily make it faster, if a model was trained with a certain size it may predict faster at that size (e.g. OSCF/TrapperAI-v02.2024 predicts at 40+ iterations per second when resized to 640 and ~31 when left to its default 1024pd)

  • Resizing (if provided a single int, not a tuple) works by making the larger side of the image equal to the given one, if padding is needed grey is used(?)

• Half-life precision (if supported by GPU) is really cool! half=True makes stuff faster (no idea about prediction quality yet)

  • And batch size obviously

• vid_stride predicts every Nth video frame, was almost going to write that myself

All-in-all I like ultralytics/YOLO

Assuming you’re doing a YOLO run w/

yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=500 imgsz=640 device=0,1 name=yolo11s-aug-500epochs-full

• name there becomes training run name in wandb + directory name in /data/projct
  • (project on wandb will be `-data-project-)
• To enable wandb:

pip install wandb
yolo settings wandb=True
wandb login 

Or if you’re inside an ultralytics:latest Docker container,

apt install -y bash screen
bash
pip install wandb
yolo settings wandb=True
wandb login 
screen 
yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=500 imgsz=640 device=0,1 name=yolo11s-aug-500epochs-full

Also useful:

# get a model file
wandb artifact get /proje:ject/run_alxxxpy7_model:v0 --root target_director/

TL;DR:
git clean -d -f .

If a broken clone / switch leaves stray broken files: error: The following untracked working tree files would be overwritten by checkout:, that fixes it.¹

I had exotic not enough shared memory crashes, ty GC for giving me these lines I do not yet understand but that seem to work, later I’ll dig into why (TODO)

apiVersion: v1
kind: Pod
metadata:
  name: CHANGEME
  namespace: CHANGEME-ns
spec:
  restartPolicy: Never
  containers:
    - name: sh-temp-yolo-container-3
      image: ultralytics/ultralytics:latest
      command: ["/bin/sh", "-c"]
      args: 
        - "yolo detect train model=yolo11s.pt data=/data/data/data.yaml project=/data/project/ epochs=30 imgsz=640 device=0,1"
      resources:
        requests:
          nvidia.com/gpu: "2" # GPUs for each training run
          ephemeral-storage: "12Gi"
        limits:
          nvidia.com/gpu: "2" # same as requests nvidia.com/gpu
          ephemeral-storage: "14Gi"
      volumeMounts: # Mount the persistent volume
        - name: data
          mountPath: /data
        - name: shared-memory
          mountPath: /dev/shm
  volumes: 
    - name: shared-memory
      emptyDir:
        medium: Memory
    - name: data
      persistentVolumeClaim:
        claimName: sh-temp-yolo-pvc

Both requests AND limits, as well as mount shared memory in volumeMounts + volumes.

For one-off HTML exports, found the plugin KosmosisDire/obsidian-webpage-export: Export html from single files, canvas pages, or whole vaults. Direct access to the exported HTML files allows you to publish your digital garden anywhere. Focuses on flexibility, features, and style parity.

It exports both the vault and individual pages, and adds things like toc on the left and toggles and optionally file browsing. Much better than the other pandoc-based export plugin that I could not get to work reliably for exporting good-looking HTML

• Qutebrowser:
  :set colors.webpage.darkmode.enabled true Really neat actually! ALSO: colors.webpage.preferred_color_scheme: dark tells websites my preference
• Nvim: :colorscheme zaibatsu
• Redshift: redshift -r -P -O 4000 -b 0.3
• If I’m doing all that, I probably want to mute my speakers as well

Nested list comprehensions are a horrible idea because they are hard to parse, and I never understood them, BUT.¹

python - How do I make a flat list out of a list of lists? - Stack Overflow has a discussion in the accepted answer about the suggested syntax to flatten lists, and I get it now.

flat_list = [
    x
    for xs in xss
    for x in xs
]

# equivalent to 
flat_list = []

for xs in xss:
    for x in xs:
        flat_list.append(x)

So,

[x for xs in xss for x in xs]

Comments:

I found the syntax hard to understand until I realized you can think of it exactly like nested for loops. for sublist in l: for item in sublist: yield item

[leaf for tree in forest for leaf in tree]

I kept looking here every time I wanted to flatten a list, but this gif is what drove it home: i.sstatic.net/0GoV5.gif 

GIF IN QUESTION, after which it clicked for me:

The first element is the one that gets returned!

for tree in forest: for leaf in tree: return leaf
[leaf (for tree in forest, for leaf in tree)]
[leaf (for tree in forest for leaf in tree)]
[leaf for tree in forest for leaf in tree]

Found Understanding nested list comprehension syntax in Python — /var/ which expands on this, quoting PEP

It is proposed to allow conditional construction of list literals using for and if clauses. They would nest in the same way for loops and if statements nest now.

It then shows:

for x in non_flat:
    if len(x) > 2
        for y in x:
            y
# equivaent to
>>> [ y for x in non_flat if len(x) > 2 for y in x ]

MIND. BLOWN.

I’m not sure “this requires you to understand Python syntax” is an argument against using a given technique in Python This is about itertools.chain(*list, which is the way to go imo. But still, * is python syntax, otherwise there are more or less readable ways to do thigs and nested comprehensions are rarely worth it

TL;DR clone to unencrypted directory

error: unable to create file datasets/processed/GitHub-Mensch-Animal_Finetuned/data/val/labels/1713256557366,hintergrund-meister-lampe-geht-das-licht-aus-vom-rueckgang-der-arten-tierische-und-pflanzliche-neubuerger-108~v-16x9@2dM-ad6791ade5eb8b5c935dd377130b903c4b5781d8.txt: File name too long

error: cannot stat ‘datasets/processed/GitHub-Mensch-Animal_Finetuned/data/val/images/1713256557366,hintergrund-meister-lampe-geht-das-licht-aus-vom-rueckgang-der-arten-tierische-und-pflanzliche-neubuerger-108~v-16x9@2dM-ad6791ade5eb8b5c935dd377130b903c4b5781d8.jpg’: File name too long

The usual solution¹ is to set longpaths = true in git config or during clone (git clone -c core.longpaths=true <repo-url>)

Didn’t solve this for me.

BUT apparently my encrypted $HOME has something to do with this, because filenames can get longer (?) in this case and go over the limit?.. git checkout-index: unable to create file (File name too long) - Stack Overflow

And one solution is to clone to /tmp or whatever is not encrypted by encryptfs.

(And in my case I could rename these files in a commit in /tmp and after that it worked, as long as I don’t check out the revisions with the long filenames)

• Goal: centerline trace pngs
• Inscape
  • Script, extension, now part of Inkscape: inkscape-centerline-trace/centerline-trace.py at master · fablabnbg/inkscape-centerline-trace
  • Inkscape as a purely command line tool : r/Inkscape reddit on using the CLI and running the extension standalone and how it doesn’t really work
  • Can’t get it to work using CLI either, there’s just bitmap-trace that has no centerline option
• Not inkscape
  • autotrace/autotrace: bitmap to vector graphics converter
  • Potrace
    • no centerline

Autotrace is awesome!

This alone works really nicely:

autotrace -centerline AMPERSAND.png  -output-file AMPERSAND.svg

Fish script for batch processing, courtesy of ChatGPT:

#!/usr/bin/fish

# Check if autotrace is installed
if not type -q autotrace
    echo "autotrace is not installed. Please install it first."
    exit 1
end

# Loop through each .png file provided as an argument
for file in $argv
    # Check if the file extension is .png
    if string match -r '\.png$' $file
        # Set the output filename by replacing .png with .svg
        set output_file (string replace -r '\.png$' '.svg' $file)

        # Execute autotrace with centerline option
        autotrace -centerline $file -output-file $output_file

        # Confirmation message
        echo "Processed $file to $output_file"
    else
        echo "Skipping $file: not a .png file"
    end
end

And a more simple one:

#!/usr/bin/fish
for file in $argv
    autotrace -centerline $file -output-file "$file.svg"
end

Options

ChatGPT says this:

autotrace -centerline -input-format png -output-format svg -output-file traced_dejavu.svg -dpi 300 -error-threshold 0.5 -corner-threshold 85 -filter-iterations 2 -noise-removal 0.99 -line-threshold 0.5 -corner-surround 3

(et 1 is best)

Using the Command Line - Inkscape Wiki

• inkscape action-list shows all available actions

• man inkscape is the latest and best

• inkscape AMPERSAND.png --export-type="svg" --export-area-page --batch-process works but asks me about import options

inkscape --shell, man page gives examples:

file-open:file1.svg; export-type:pdf; export-do; export-type:png; export-do
                   file-open:file2.svg; export-id:rect2; export-id-only; export-filename:rect_only.svg; export-do

OK this works for no questions about how to import it:

> file-open:AMPERSAND.png
> export-filename:AM.svg
> export-do