serhii.net

The Open LLM Leaderboard is dead¹, as good time as any to look for new eval stuff!

• HF universe

  • The Open LLM Leaderboard people are actually OpenEvals (OpenEvals), and they created other cool stuffs
    • huggingface/lighteval: Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends eval suite
      • I like their documentation for new task / new model
    • huggingface/evaluation-guidebook: Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
      • the contents are awesome, w/ examples, model-as-a-judge, etc.
  • evaluate-metric (Evaluate Metric) recommended by them as a guide to existing metrics

• Harnesses

  • Evalverse: Unified and Accessible Library for Large Language Model Evaluation meta-thing that can run different harnesses based on the target. Github repo archived? UpstageAI/evalverse: The Universe of Evaluation. All about the evaluation for LLMs.
  • open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
    • don’t really like their documentation on adding datasets/models
  • The venerable stanford-crfm/helm: Holistic Evaluation of Language Models (HELM)
  • openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

• Resources / articles:

  • How to evaluate by Meta: Validation | How-to guides
  • HF eval guidebook: huggingface/evaluation-guidebook

Backup and Restore | Gitea Documentation has the full detailed story.

The easy stupid way for backing up gitea running in docker, untested and will fail if DB was being used during dumping. I haven’t tried to import it anywhere as well, may be wrong.

docker exec -it --user git gitea-container bash

gitea dump

# then outside the container, copy from the gitea container to host OSooj

docker cp gitea-container:/whatveer/gitea-dump.zip /tmp

• M.2 slots have keys: How do I install an M.2 SSD on my computer? - Transcend Information, Inc.
• B+M means both B and M slots are acceptable for the B+M module

CLI: sudo lshw -C disk tells you all disks h

In fsspec fs.copy() doesn’t really work from local to remote, also existing or not-existing directories etc.

Their documentation has a whole page on this: Copying files and directories — fsspec 2024.10.0.post13+gdbed2ec.d20241115 documentation

GitLab CLI - glab | GitLab Docs: CLI thingy to interact with gitlab.

TL;DR: glab ci status or whatever.

It’s really neat and has a cool CLI interface, either you set things through flags or you get a neat menu to choose from!

Smart enough to parse current directory!

Git Tag: A Tutorial for Tagging Releases in Git - DEV Community

#ligthweight tag
git tag v1.0.0

# full
git tag -a v1.0.0 -m "Releasing version v1.0.0"

Tags don’t get pushed automatically. For this, git push origin v1.0.0

• Writing your pyproject.toml - Python Packaging User Guide
• uv uses hatch which has this to say about dynamic versioning: Configuring project metadata - Hatch

[project] 
dynamic = ["version"] 

[tool.hatch.version] 
path = "..."

Path is a python file w/ version info. If using src layout, src has to be included in the path.j

For uv, this works. Described for example here: Versioning Python Projects with Hatch (I like __init__.py though, not about as that guide does)

uv-dynamic-versioning · PyPI exists but I don’t really see why.

Because every single goddamn time

Command line: How do you rotate a PDF file 90 degrees? - Unix & Linux Stack Exchange

pdftk input.pdf cat 1-endwest output output.pdf

1- is needed because page range, here for all pages.

endwest etc from man page:

 [<begin page number>[-<end page number>[<qualifier>]]][<page rotation>]

The qualifier can be even or odd, and the page rotation can be north, south, east, west, left, right, or down.
Each option sets the page rotation as follows (in degrees): north: 0, east: 90, south: 180, west: 270, left: -90, right: +90, down: +180. left, right, and down make relative adjustments to a page’s rotation.

• Just got sold on this as a way to automate creation of config maps, secrets and friends!
  • Though allegedly hard-ish to learn
• Need to look deeper.

This is brilliant: Git, rewrite previous commit usernames and emails - Stack Overflow

TL;DR

git config --global alias.change-commits '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f"

Then

git change-commits GIT_AUTHOR_NAME "old name" "new name"

# last 10 commits
git change-commits GIT_AUTHOR_EMAIL "old@email.com" "new@email.com" HEAD~10..HEAD

Depending on why I need this, I may need also GIT_COMMITTER_[NAME/EMAIL]

For multiple times, I created an change-commit-f that forces overwiriting the backup:

git config --global alias.change-commits-f '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch -f --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f"

A quick tldr from other answer, may be better but untested:

git config alias.change-commits '!'"f() { VAR=\$1; OLD=\$2; NEW=\$3; shift 3; git filter-branch --env-filter \"if  \\\"\$\`echo \$VAR\`\\\" = '\$OLD' ; then export \$VAR='\$NEW'; fi\" \$@; }; f "
git change-commits GIT_AUTHOR_NAME "<Old Name>" "<New Name>" -f
git change-commits GIT_AUTHOR_EMAIL <old@email.com> <new@email.com> -f
git change-commits GIT_COMMITTER_NAME "<Old Name>" "<New Name>" -f
git change-commits GIT_COMMITTER_EMAIL <old@email.com> <new@email.com> -f

(Previously: 220408-1822 Gitlab ‘you cannot push commits for ..’ error)