serhii.net

python - Split / Explode a column of dictionaries into separate columns with pandas - Stack Overflow taught be about pandas.json_normalize — pandas 2.1.1 documentation:

In: json-like (dict, list, ..) Out: pandas dataframe!

Name

• evalUAtion
• as package:
  • eval-UA-tion / eval_UA_tion
• For SEO:
  • eval-UA-tion Ukrainian LM benchmark

Logo

• Clearly highlight the UA
• make it include the - if they end up in the name
• obvious два кольори мої BUT ALSO
  • red and black and pixel-art faux-vyshyvanka?
  • CAN USE THE ONE BN GAVE ME?

Ideas

• vyshyvanka:
  • ’eval-UA-tion’ with a typical UA ornament near one of the sides of the U, and a small logo would contain it together with an U
• петриківський розпис
  • так само, на одній з частин U
• гілочка калини

UA ornaments

• Символічне значення традиційних орнаментів українських вишитих рушників has this and I love it:
• This has a lot of reference photos but none usable: 12 головних символів української вишивки
• Текстова вишивка з категорії Інше | Ornament Name
• Вишиванки | МореСхем

Tools

• Онлайн-конструктор українського орнаменту. Конструктор схем для вишивки хрестиком. Конструктор схем для бісероплетіння

Logo iterations

1

• I like eval-UA-tion since it works both with colors and as plain monotype text!
• I don’t like the overused y/b colors

2

Some drafts I did in inkscape:

And just for fun:

2023-12-13 19:23

ChatGPT generated this:

It’s internal prompt for the picture, based on inspect element, was alt="Logo design for 'eval-UA-tion', a benchmark for Ukrainian language models. Incorporate the word 'eval-UA-tion' in a stylish font, with a sunflower replacing the letter 'o'. Add elements that give a Ukrainian touch, such as traditional Ukrainian patterns or colors (blue and yellow). The design should be modern, clear, and professional, suitable for a technical and academic setting."

# 2 after comma
pd.set_option("display.precision", 2)
# Suppress scientific notation
pd.options.display.float_format = "{:.0f}".format
# for more natural 100,233.23-like output
pd.options.display.float_format = "{:,.3f}".format

Setting as a context¹:

with pd.option_context('display.float_format', lambda x: f'{x:,.3f}'):
    display(df.describe())

Also: I can format a float column (’temporarily’) not just how I always did, but also in a way simpler way²:

# before
ds["percent"].apply(lambda x: f"{x:.2%}")
# after
ds["percent"].apply("{:.2%}".format)

I forgot you can do "string".format(variable)!

Also TIL display() for jupyter-notebooks when it’s not the return value (e.g. if you’re exiting a context, df.describe() alone there would not have shown the description)

One way to do it, if it’s all for all:

df.groupby("collection")[
    ["num_pages", "num_chars", "num_tokens", "num_sentences"]
].agg(
    [
        # "count",
        "sum",
        "mean",
        # "std",
    ]
)

An even better way:

# ...
].agg(
    num_documents=("num_pages", "count"),
    num_pages=("num_pages", "sum"),
    mean_pages=("num_pages", "mean"),
    mean_tokens=("num_tokens", "mean"),
)

They are literally named tuples! Yay for Named Aggregation¹!

Draft.

Context: 230529-2208 Seaborn matplotlib labeling data points

Given: need to make the limits larger to fit text, the last lines here:

data =  df_pages.reset_index().sort_values('num_pages')

ax = sns.barplot(data,y="collection",x="num_pages")

# label points
for i in ax.axes.containers:
    ax.bar_label(
        i,
    )

# make the labels fit the limits
xlim = ax.axes.get_xlim()[1]
new_xlim = xlim + 14600

ax.axes.set_xlim(0, new_xlim)

Question: by how much?

Answer:

• Transformations Tutorial — Matplotlib 3.8.0 documentation for converting text-pixels to data-limit
• ??? to get the text dimensions

for i in ax.axes.containers:
    an = ax.bar_label(
        i,
    )

# `an` is a list of all Annotations

an[0].get_window_extent()

>>> Bbox(88.66956472198585, 388.99999999999994], [123.66956472198585, 402.99999999999994)

def get_text_size(anno):  # Annotation
    """ TODO: get array of annos, find the leftmost one etc."""
    bbox = anno.get_window_extent()
    ext = bbox.bounds
	# > (91.43835300441604, 336.19999999999993, 35.0, 14.0)
    x=ext[2]
    y=ext[3]
    return x,y

"""
ano = an[1]
bbox = ano.get_window_extent()
bbox.bounds
> (91.43835300441604, 336.19999999999993, 35.0, 14.0)
"""
get_text_size(an[6])

Gitlab introduced tasks, and they get shown by default in the issue list. Type != task in the search leaves only the issues.

Can one save search templates?..

Is this needed or I can just use one of the existing ones? I’ll use one of the existing ones!

Then this is about notes about choosing one and adapting my own tasks for it.

First of all, I’d like the generator things to be runnable through Docker, especially the pravda crawler!

Related:

• 230928-1735 Other LM Benchmarks notes has examples
• I have to support prompts, e.g. see what other papers¹ do, as well as the harnesses

General:

• I don’t have to forget that the OpenAI API exists!
  • And I can use it for evaluation too!

Other / useful libraries:

• 230928-1735 Other LM Benchmarks notes has a list of harnesses, todo move here.
• bigscience-workshop/lm-evaluation-harness: A framework for few-shot evaluation of autoregressive language models.
• TheoremOne/llm-benchmarker-suite: LLM Evals Leaderboard

Main idea

• Following established practice, dataset on HF hub and some magic code to convert it into actual LM inputs.
• Have a versioning system (both for individual tasks and the benchmark in general?)

Architecture

Tasks

• A task has a metadata file, and the task data.
• task metadata
  • VERSION!
  • Can contain a task description for the model as prompt
  • Can contain the format-strings etc. for building examples
• The task data
  • can be either a
    • .json
    • a HF dataset string
  • contains:
    • All the data needed to build test cases for each example

Task types

• exact match: true/false

• multiple choice

  • incl. binary yes/no

• lmentry/lmentry/predict.py at main · aviaefrat/lmentry contains the predicting code used to evaluate it using different kinds of models - I’ll need this.

Links

• HF evaluation library
• lmentry
  • lmentry/lmentry/constants.py at main · aviaefrat/lmentry
  • lmentry/lmentry/predict.py at main · aviaefrat/lmentry
• UA models: Models - Hugging Face
  • I should look into QA models as well!

SWAG seems the closest out of the modern models to UA-CBT — one-word completions etc. I should look into what exactly they do

NarrativeQA!

Literature review

LessWrong

• Literature Review For Academic Outsiders: What, How, and Why — LessWrong

• ‘Literature review’ the process is a way to become familiar with what work has already been done in a particular field or subject by searching for and studying previous work

• Every time I do research I perform a simple thought experiment: assuming somewhere in the world exists evidence that would prove or disprove my hypothesis, where is it?

• Citations are a hierarchy of ideas

• Scholarship: How to Do It Efficiently — LessWrong

Style etc.

My old note about tenses in a bachelor thesis: Day 155 - serhii.net linking to the excellent Effective Writing | Learn Science at Scitable

• My notes at 231201-1401 How to read and write a paper according to hackernews. Main takeaway for me is to keep in mind my target audience, what they know and what they don’t, when writing.

Grammar glossing

Leipzig glossing rules

Leipzig Glossing rules seems to be the key for me:

• Dept. of Linguistics | Resources | Glossing Rules

• List of glossing abbreviations - Wikipedia

• Markdown and python and stuff

• Markdown

  • cysouw/pandoc-ling: Pandoc Lua filter for linguistic examples
    • Using pandoc-ling
  • gunnarnl/pangb4e: Pandoc filter for gb4e support
    • Ref: Gunnar Lund - Markdown for linguists
  • parryc/doctor_leipzig: Leipzig, MD - glossing for Markdown - one can always do <span style="font-variant:small-caps;">Hello World</span>¹

• Python

  • Tutorial — pyigt 2.1.1.dev0 documentation

Linguistics basics

• Random book I found: Essentials of Linguistics, 2nd edition – Simple Book Publishing
• Also
  • (172) What are some books I can read if I want to get into studying linguistics casually? : linguistics
    • Steven Pinker’s “The Language Instinct” - allegedly really cool but not for my purposes

Essentials of Linguistics, 2nd edition

The online version¹ has cool tests at the end!

Generally: a lot of it is about languages/power, indigenous languages etc. Might be interesting for me wrt. UA/RU and colonialism

• Chapter 5 / Morphology gets interesting
  • 5.7 Inflectional morphology!
• 6: Syntax - even more interesting
  • 6.2 word order
  • p.264 Key grammatical terminology
  • word order
  • really cool and quite technical up until the end, esp. trees
• 7. Semantics
• 8. Pragmatics
  • todo - all of it

https://peps.python.org/pep-0673/

from typing import Self

class Shape:
    def set_scale(self, scale: float) -> Self:
        self.scale = scale
        return self

Related: 220726-1638 Python typing classmethods return type

I remember writing about the typevar approach but cannot find it…