serhii.net

Had PDF files, extracted text with Pymupdf, in some of the output txts I had weird strings:

# sometimes with real chars mixed in
�������#&'���()��"#��*����������%�
# sometimes - often - not
������ ������

Tried to understand what the “�” actually are, guess the encoding etc. Encoding was always utf8, according to python chardet and debian uchardet.

Remembered and tried CyberChef, it returned it all as identical repeating code points.

hexdump showed me that they actually ARE repeating code points!

Remembered vim can do this - it can¹ - vim’s g8 binding for the same, as well as :as to show info about the char under the cursor, confirmed it - it’s all one character, specifically (:as) ef bf bd.

I googled that string, found² that it’s Unicode Character ‘REPLACEMENT CHARACTER’ (U+FFFD).

Basically it’s when input is not valid UTF8, and we replace the character with that symbol. The original characters are lost.

Python’s unicodedata has unicodedata.name() that returns directly 'REPLACEMENT CHARACTER'.

This explains why all the character detection bits said utf-8 - it was utf-8 characters, the exact same one in fact, haha.

python - Split / Explode a column of dictionaries into separate columns with pandas - Stack Overflow taught be about pandas.json_normalize — pandas 2.1.1 documentation:

In: json-like (dict, list, ..) Out: pandas dataframe!

Name

• evalUAtion
• as package:
  • eval-UA-tion / eval_UA_tion
• For SEO:
  • eval-UA-tion Ukrainian LM benchmark

Logo

• Clearly highlight the UA
• make it include the - if they end up in the name
• obvious два кольори мої BUT ALSO
  • red and black and pixel-art faux-vyshyvanka?
  • CAN USE THE ONE BN GAVE ME?

Ideas

• vyshyvanka:
  • ’eval-UA-tion’ with a typical UA ornament near one of the sides of the U, and a small logo would contain it together with an U
• петриківський розпис
  • так само, на одній з частин U
• гілочка калини

UA ornaments

• Символічне значення традиційних орнаментів українських вишитих рушників has this and I love it:
• This has a lot of reference photos but none usable: 12 головних символів української вишивки
• Текстова вишивка з категорії Інше | Ornament Name
• Вишиванки | МореСхем

Tools

• Онлайн-конструктор українського орнаменту. Конструктор схем для вишивки хрестиком. Конструктор схем для бісероплетіння

Logo iterations

1

• I like eval-UA-tion since it works both with colors and as plain monotype text!
• I don’t like the overused y/b colors

2

Some drafts I did in inkscape:

And just for fun:

2023-12-13 19:23

ChatGPT generated this:

It’s internal prompt for the picture, based on inspect element, was alt="Logo design for 'eval-UA-tion', a benchmark for Ukrainian language models. Incorporate the word 'eval-UA-tion' in a stylish font, with a sunflower replacing the letter 'o'. Add elements that give a Ukrainian touch, such as traditional Ukrainian patterns or colors (blue and yellow). The design should be modern, clear, and professional, suitable for a technical and academic setting."

# 2 after comma
pd.set_option("display.precision", 2)
# Suppress scientific notation
pd.options.display.float_format = "{:.0f}".format
# for more natural 100,233.23-like output
pd.options.display.float_format = "{:,.3f}".format

Setting as a context¹:

with pd.option_context('display.float_format', lambda x: f'{x:,.3f}'):
    display(df.describe())

Also: I can format a float column (’temporarily’) not just how I always did, but also in a way simpler way²:

# before
ds["percent"].apply(lambda x: f"{x:.2%}")
# after
ds["percent"].apply("{:.2%}".format)

I forgot you can do "string".format(variable)!

Also TIL display() for jupyter-notebooks when it’s not the return value (e.g. if you’re exiting a context, df.describe() alone there would not have shown the description)

One way to do it, if it’s all for all:

df.groupby("collection")[
    ["num_pages", "num_chars", "num_tokens", "num_sentences"]
].agg(
    [
        # "count",
        "sum",
        "mean",
        # "std",
    ]
)

An even better way:

# ...
].agg(
    num_documents=("num_pages", "count"),
    num_pages=("num_pages", "sum"),
    mean_pages=("num_pages", "mean"),
    mean_tokens=("num_tokens", "mean"),
)

They are literally named tuples! Yay for Named Aggregation¹!

Draft.

Context: 230529-2208 Seaborn matplotlib labeling data points

Given: need to make the limits larger to fit text, the last lines here:

data =  df_pages.reset_index().sort_values('num_pages')

ax = sns.barplot(data,y="collection",x="num_pages")

# label points
for i in ax.axes.containers:
    ax.bar_label(
        i,
    )

# make the labels fit the limits
xlim = ax.axes.get_xlim()[1]
new_xlim = xlim + 14600

ax.axes.set_xlim(0, new_xlim)

Question: by how much?

Answer:

• Transformations Tutorial — Matplotlib 3.8.0 documentation for converting text-pixels to data-limit
• ??? to get the text dimensions

for i in ax.axes.containers:
    an = ax.bar_label(
        i,
    )

# `an` is a list of all Annotations

an[0].get_window_extent()

>>> Bbox(88.66956472198585, 388.99999999999994], [123.66956472198585, 402.99999999999994)

def get_text_size(anno):  # Annotation
    """ TODO: get array of annos, find the leftmost one etc."""
    bbox = anno.get_window_extent()
    ext = bbox.bounds
	# > (91.43835300441604, 336.19999999999993, 35.0, 14.0)
    x=ext[2]
    y=ext[3]
    return x,y

"""
ano = an[1]
bbox = ano.get_window_extent()
bbox.bounds
> (91.43835300441604, 336.19999999999993, 35.0, 14.0)
"""
get_text_size(an[6])

Gitlab introduced tasks, and they get shown by default in the issue list. Type != task in the search leaves only the issues.

Can one save search templates?..

Is this needed or I can just use one of the existing ones? I’ll use one of the existing ones!

Then this is about notes about choosing one and adapting my own tasks for it.

First of all, I’d like the generator things to be runnable through Docker, especially the pravda crawler!

Related:

• 230928-1735 Other LM Benchmarks notes has examples
• I have to support prompts, e.g. see what other papers¹ do, as well as the harnesses

General:

• I don’t have to forget that the OpenAI API exists!
  • And I can use it for evaluation too!

Other / useful libraries:

• 230928-1735 Other LM Benchmarks notes has a list of harnesses, todo move here.
• bigscience-workshop/lm-evaluation-harness: A framework for few-shot evaluation of autoregressive language models.
• TheoremOne/llm-benchmarker-suite: LLM Evals Leaderboard

Main idea

• Following established practice, dataset on HF hub and some magic code to convert it into actual LM inputs.
• Have a versioning system (both for individual tasks and the benchmark in general?)

Architecture

Tasks

• A task has a metadata file, and the task data.
• task metadata
  • VERSION!
  • Can contain a task description for the model as prompt
  • Can contain the format-strings etc. for building examples
• The task data
  • can be either a
    • .json
    • a HF dataset string
  • contains:
    • All the data needed to build test cases for each example

Task types

• exact match: true/false

• multiple choice

  • incl. binary yes/no

• lmentry/lmentry/predict.py at main · aviaefrat/lmentry contains the predicting code used to evaluate it using different kinds of models - I’ll need this.

Links

• HF evaluation library
• lmentry
  • lmentry/lmentry/constants.py at main · aviaefrat/lmentry
  • lmentry/lmentry/predict.py at main · aviaefrat/lmentry
• UA models: Models - Hugging Face
  • I should look into QA models as well!

SWAG seems the closest out of the modern models to UA-CBT — one-word completions etc. I should look into what exactly they do

NarrativeQA!

Linguistics basics

• Random book I found: Essentials of Linguistics, 2nd edition – Simple Book Publishing
• Also
  • (172) What are some books I can read if I want to get into studying linguistics casually? : linguistics
    • Steven Pinker’s “The Language Instinct” - allegedly really cool but not for my purposes

Essentials of Linguistics, 2nd edition

The online version¹ has cool tests at the end!

Generally: a lot of it is about languages/power, indigenous languages etc. Might be interesting for me wrt. UA/RU and colonialism

• Chapter 5 / Morphology gets interesting
  • 5.7 Inflectional morphology!
• 6: Syntax - even more interesting
  • 6.2 word order
  • p.264 Key grammatical terminology
  • word order
  • really cool and quite technical up until the end, esp. trees
• 7. Semantics
• 8. Pragmatics
  • todo - all of it

https://peps.python.org/pep-0673/

from typing import Self

class Shape:
    def set_scale(self, scale: float) -> Self:
        self.scale = scale
        return self

Related: 220726-1638 Python typing classmethods return type

I remember writing about the typevar approach but cannot find it…