serhii.net

Goal: find identical words with diff embeddings in RU and UA, use that to generate examples.

• Alignment
  • models.translation_matrix – Translation Matrix model — gensim was my initial idea
• Embeddings
  • Models: lang-uk - Ukrainian
  • osyvokon/awesome-ukrainian-nlp: Curated list of Ukrainian natural language processing (NLP) resources (corpora, pretrained models, libriaries, etc.) links to FastText
    • babylonhealth/fastText_multilingual: Multilingual word vectors in 78 languages DAMN

FastText

Link broken but I think I found the download page for the vectors

Their blog is also down but they link the howto from the archive Aligning vector representations – Sam’s ML Blog

Download: fastText/docs/crawl-vectors.md at master · facebookresearch/fastText

axel https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.uk.300.bin.gz
axel https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.ru.300.bin.gz

It’s taking a while.

EDIT: Ah damn, had to be the text ones, not bin. :( starting again

EDIT2: THIS is the place: fastText/docs/pretrained-vectors.md at master · facebookresearch/fastText

https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.uk.vec
https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.ru.vec

UKR has 900k lines, RUS has 1.8M — damn, it’s not going to be easy.

What do I do next, assuming this works?

Other options

• Pretrained Models — Sentence-Transformers documentation for sentences
  • I like the bit about measuring semantic textual similarity between langs
• NLPL word embeddings repository
  • includes models with smaller vector sizes, but I’ll have to align them myself

Next steps

Assuming I found out that RU-кит is far in the embedding space from UKR-кіт, what do I do next?

How do I test for false friends?

Maybe these papers about Surzhyk might come in handy now, especially <_(@Sira2019) “Towards an automatic recognition of mixed languages: The Ukrainian-Russian hybrid language Surzhyk” (2019) / Nataliya Sira, Giorgio Maria Di Nunzio, Viviana Nosilia: z / http://arxiv.org/abs/1912.08582 / _>.

• Existing research
  • false friends nlp machine learning - Google Scholar damn, apparently everyone wrote papers about identifying false friends
    • But not for RU-UA it seems!
    • <_(@inkpen2005automatic) “Automatic identification of cognates and false friends in French and English” (2005) / Diana Inkpen, Oana Frunza, Grzegorz Kondrak: z / / _> mentions different types of cognates, false friends being only one of them
      • PARTIAL COGNATES where they have the same meaning in some but not all contexts!
        • but that paper doesn’t identify them
        • Recherche uO Research: Automatic identification of cognates, false friends, and partial cognates is a master’s thesis about that from 2005
        • Automated identification of borrowings in multilingual wordlists :: MPG.PuRe just as interesting! 2021
        • partial cognates automated identification - Google Scholar is a better search string for more interesting papers
• Generating examples
  • GPT4 can generate sentences using a specific word
  • I can look for specific sentences in corpora, but I’d have the usual google problem
  • I can use pymorphy3 etc. as usual to make it more interesting, BUT the issue with picking the correct parsing stands

Back to python

Took infinite time & then got killed by Linux.

from fasttext import FastVector
#  ru_dictionary = FastVector(vector_file='wiki.ru.vec')
ru_dictionary = FastVector(vector_file='/home/sh/uuni/master/code/ru_interference/DATA/wiki.ru.vec')
uk_dictionary = FastVector(vector_file='/home/sh/uuni/master/code/ru_interference/DATA/wiki.uk.vec')

uk_dictionary.apply_transform('alignment_matrices/uk.txt')
ru_dictionary.apply_transform('alignment_matrices/ru.txt')

print(FastVector.cosine_similarity(ua_dictionary["кіт"], ru_dictionary["кот"]))

Gensim it is.

To load:

from gensim.models import KeyedVectors
from gensim.test.utils import datapath

ru_dictionary = 'DATA/small/wiki.ru.vec'
uk_dictionary = 'DATA/small/wiki.uk.vec'

model_ru = KeyedVectors.load_word2vec_format(datapath(ru_dictionary))
model_uk = KeyedVectors.load_word2vec_format(datapath(uk_dictionary))

Did ru_model.save(...) and then I can load it as >>> KeyedVectors.load("ru_interference/src/ru-model-save")

Which is faster — shouldn’t have used the text format, but that’s on me.

 from gensim.models import TranslationMatrix
tm = TranslationMatrix(model_ru,model_uk, word_pairs)

(Pdb++) r = tm2.translate(ukrainian_words,topn=3)
(Pdb++) pp(r)
OrderedDict([('сонце', ['завишня', 'скорбна', 'вишня']),
             ('квітка', ['вишня', 'груша', 'вишнях']),
             ('місяць', ['любить…»', 'гадаю…»', 'помилуй']),
             ('дерево', ['яблуко', '„яблуко', 'яблуку']),
             ('вода', ['вода', 'риба', 'каламутна']),
             ('птах', ['короваю', 'коровай', 'корова']),
             ('книга', ['читати', 'читати»', 'їсти']),
             ('синій', ['вишнях', 'зморшках', 'плакуча'])])

OK, then definitely more words would be needed for the translation.

Either way I don’t need it, I need the space, roughly described here: mapping - How do i get a vector from gensim’s translation_matrix - Stack Overflow

• Oh this is even better: gbnlp/nlp3.ipynb at master · ShadarRim/gbnlp
  • dicts! nikhovas/ml-labs-bad-data-fix

Next time:

• get more words, e.g. from a dictionary

• get a space

• play with translations

• python - Combining/adding vectors from different word2vec models - Stack Overflow mentions transvec · PyPI that allows accessing the vectors

  • But it puts both in the same vector space, without differentiating. Which is bad for me, since I need to keep them separate and some words are identical.
  • But maybe I can reuse the translation matrix

Vector blues

• Jpsaris/transvec: Translate word embeddings across models fixes the things I wanted to fix myself in the original implementation w/ new gensim version — note to self, forking things is allowed and is better than editing files locally The wiki vectors are kinda garbage, with most_similar returning not semantically similar words, but ones looking like candidatte next words. And a lot of random punctuation inside the tokens. Maybe I’m oding sth wrong?

Anyway - my only reason for them was ft multilingual, I can do others now.

• I can reduce the size of the CC
  • fastText/docs/crawl-vectors.md at master · facebookresearch/fastText
  • reduced to 100 both of them
  • The CC ones work!
  • except ‘word in model’ always returns True?…
    • word in model.key_to_index (which is a dict) works
  • really nice translations!
• Transvec

*** RuntimeError: scikit-learn estimators should always specify their parameters in the signature of their __init__ (no varargs). <class 'transvec.transformers.TranslationWordVectorizer'> with constructor (self, target: 'gensim.models.keyedvectors.KeyedVectors', *sources: 'gensim.models.keyedvectors.KeyedVectors', alpha: float = 1.0, max_iter: Optional[int] = None, tol: float = 0.001, solver: str = 'auto', missing: str = 'raise', random_state: Union[int, numpy.random.mtrand.RandomState, NoneType] = None) doesn't  follow this convention.

ah damn. Wasn’t an issue with the older one, though the only thing that changed is https://github.com/big-o/transvec/compare/master...Jpsaris:transvec:master

So

Decided to leave this till better times, but play with this one more hour today.

Coming back to mapping - How do i get a vector from gensim’s translation_matrix - Stack Overflow, I need mapped_source_space.

I should have used pycharm at a much earlier stage in the process.

• mapped_source_space contains a matrix with the 4 vectors mapped to the target space.
• A Space is a matrix w/ vectors, and the dicts that tell you which word is where.
• For my purposes, I can ’translate’ the interesting (to me) words and then compare their vectors to the vectors of the corresponding words in the target space.

Why does source_space have 1.8k words, while the source embedding space has 200k?

Ah, tmp.translate() can translate words not found in source space. Interesting!

AHA - source/target space gets build only based on the words provided for training, 1.8k in my case. Then it builds the translation matrix based on that.

BUT in translate() the target matrix gets build based on the entire vector!

Which means:

• for rus/source words, I can just use the word in the original rus embedding space, not tm’s source_space.
• for ukr words, I build the target space the same way

Results!

real
картошка/картопля -> 0.28
дом/дім -> 1.16
чай/чай -> 1.17
паспорт/паспорт -> 0.40
зерно/зерно -> 0.46
нос/ніс -> 0.94

false
неделя/неділя -> 0.34
город/город -> 0.35
он/он -> 0.77
речь/річ -> 0.89
родина/родина -> 0.32
сыр/сир -> 0.99
папа/папа -> 0.63
мать/мати -> 0.52

Let’s normalize:

real
картошка/картопля -> 0.64
дом/дім -> 0.64
чай/чай -> 0.70
паспорт/паспорт -> 0.72
зерно/зерно -> 0.60

false
неделя/неділя -> 0.55
город/город -> 0.44
он/он -> 0.33
речь/річ -> 0.54
родина/родина -> 0.50
сыр/сир -> 0.66
папа/папа -> 0.51
мать/мати -> 0.56

OK, so it mostly works! With good enough tresholds it can work. Words that are totally different aren’t similar (он), words that have some shared meanings (мать/мати) are closer.

Ways to improve this:

• Remove partly matching words from the list of reference translations used to build this
• Find some lists of all words in both languages
• Test the hell out of them, find the most and least similar ones

Onwards

• https://github.com/frekwencja/most-common-words-multilingual

  • https://raw.githubusercontent.com/frekwencja/most-common-words-multilingual/main/data/wordfrequency.info/ru.txt
  • https://raw.githubusercontent.com/frekwencja/most-common-words-multilingual/main/data/wordfrequency.info/uk.txt

• created pairs out of the words in the dictionaries that are identical (not кот/кіт/кит), will look at similarities of Russian word and Ukrainian word

• 422 such words in common

sorted by similarity (lower values = more fake friend-y). Nope, doesn’t make sense mostly. But rare words seem to be the most ‘different’ ones:

{'поза': 0.3139531, 'iphone': 0.36648884, 'галактика': 0.39758587, 'Роман': 0.40571105, 'дюйм': 0.43442175, 'араб': 0.47358453, 'друг': 0.4818558, 'альфа': 0.48779228, 'гора': 0.5069237, 'папа': 0.50889325, 'проспект': 0.5117553, 'бейсбол': 0.51532406, 'губа': 0.51682216, 'ранчо': 0.52178365, 'голова': 0.527564, 'сука': 0.5336818, 'назад': 0.53545296, 'кулак': 0.5378426, 'стейк': 0.54102343, 'шериф': 0.5427336, 'палка': 0.5516712, 'ставка': 0.5519752, 'соло': 0.5522958, 'акула': 0.5531602, 'поле': 0.55333376, 'астроном': 0.5556448, 'шина': 0.55686104, 'агентство': 0.561674, 'сосна': 0.56177, 'бургер': 0.56337166, 'франшиза': 0.5638794, 'фунт': 0.56592, 'молекула': 0.5712515, 'браузер': 0.57368404, 'полковник': 0.5739758, 'горе': 0.5740198, 'шапка': 0.57745415, 'кампус': 0.5792211, 'дрейф': 0.5800869, 'онлайн': 0.58176875, 'замок': 0.582287, 'файл': 0.58236635, 'трон': 0.5824338, 'ураган': 0.5841942, 'диван': 0.584252, 'фургон': 0.58459675, 'трейлер': 0.5846335, 'приходить': 0.58562565, 'сотня': 0.585832, 'депозит': 0.58704704, 'демон': 0.58801174, 'будка': 0.5882363, 'царство': 0.5885376, 'миля': 0.58867997, 'головоломка': 0.5903712, 'цент': 0.59163713, 'казино': 0.59246653, 'баскетбол': 0.59255254, 'марихуана': 0.59257627, 'пастор': 0.5928912, 'предок': 0.5933549, 'район': 0.5940658, 'статистика': 0.59584284, 'стартер': 0.5987516, 'сайт': 0.5988183, 'демократ': 0.5999011, 'оплата': 0.60060596, 'тендер': 0.6014088, 'орел': 0.60169894, 'гормон': 0.6021177, 'метр': 0.6023728, 'меню': 0.60291564, 'гавань': 0.6029945, 'рукав': 0.60406476, 'статуя': 0.6047057, 'скульптура': 0.60497975, 'вагон': 0.60551536, 'доза': 0.60576916, 'синдром': 0.6064756, 'тигр': 0.60673815, 'сержант': 0.6070389, 'опера': 0.60711193, 'таблетка': 0.60712767, 'фокус': 0.6080196, 'петля': 0.60817575, 'драма': 0.60842395, 'шнур': 0.6091568, 'член': 0.6092182, 'сервер': 0.6094157, 'вилка': 0.6102615, 'мода': 0.6106603, 'лейтенант': 0.6111004, 'радар': 0.6117528, 'галерея': 0.61191505, 'ворота': 0.6125873, 'чашка': 0.6132187, 'крем': 0.6133907, 'бюро': 0.61342597, 'черепаха': 0.6146957, 'секс': 0.6151523, 'носок': 0.6156026, 'подушка': 0.6160687, 'бочка': 0.61691606, 'гольф': 0.6172053, 'факультет': 0.6178817, 'резюме': 0.61848575, 'нерв': 0.6186257, 'король': 0.61903644, 'трубка': 0.6194198, 'ангел': 0.6196466, 'маска': 0.61996806, 'ферма': 0.62029755, 'резидент': 0.6205579, 'футбол': 0.6209573, 'квест': 0.62117445, 'рулон': 0.62152386, 'сарай': 0.62211347, 'слава': 0.6222329, 'блог': 0.6223742, 'ванна': 0.6224452, 'пророк': 0.6224489, 'дерево': 0.62274456, 'горло': 0.62325376, 'порт': 0.6240524, 'лосось': 0.6243047, 'альтернатива': 0.62446254, 'кровоточить': 0.62455964, 'сенатор': 0.6246379, 'спортзал': 0.6246594, 'протокол': 0.6247676, 'ракета': 0.6254694, 'салат': 0.62662274, 'супер': 0.6277698, 'патент': 0.6280118, 'авто': 0.62803495, 'монета': 0.628338, 'консенсус': 0.62834597, 'резерв': 0.62838227, 'кабель': 0.6293858, 'могила': 0.62939847, 'небо': 0.62995523, 'поправка': 0.63010347, 'кислота': 0.6313528, 'озеро': 0.6314377, 'телескоп': 0.6323617, 'чудо': 0.6325846, 'пластик': 0.6329929, 'процент': 0.63322043, 'маркер': 0.63358307, 'датчик': 0.6337889, 'кластер': 0.633797, 'детектив': 0.6341895, 'валюта': 0.63469064, 'банан': 0.6358283, 'фабрика': 0.6360865, 'сумка': 0.63627976, 'газета': 0.6364525, 'математика': 0.63761103, 'плюс': 0.63765526, 'урожай': 0.6377103, 'контраст': 0.6385834, 'аборт': 0.63913494, 'парад': 0.63918126, 'формула': 0.63957334, 'арена': 0.6396606, 'парк': 0.6401386, 'посадка': 0.6401986, 'марш': 0.6403458, 'концерт': 0.64061844, 'перспектива': 0.6413666, 'статут': 0.6419941, 'транзит': 0.64289963, 'параметр': 0.6430252, 'рука': 0.64307654, 'голод': 0.64329326, 'медаль': 0.643804, 'фестиваль': 0.6438755, 'небеса': 0.64397913, 'барабан': 0.64438117, 'картина': 0.6444177, 'вентилятор': 0.6454438, 'ресторан': 0.64582723, 'лист': 0.64694726, 'частота': 0.64801234, 'ручка': 0.6481528, 'ноутбук': 0.64842474, 'пара': 0.6486577, 'коробка': 0.64910173, 'сенат': 0.64915174, 'номер': 0.64946175, 'ремесло': 0.6498537, 'слон': 0.6499266, 'губернатор': 0.64999187, 'раковина': 0.6502305, 'трава': 0.6505385, 'мандат': 0.6511373, 'великий': 0.6511585, 'ящик': 0.65194154, 'череп': 0.6522753, 'ковбой': 0.65260696, 'корова': 0.65319675, 'честь': 0.65348136, 'легенда': 0.6538656, 'душа': 0.65390354, 'автобус': 0.6544202, 'метафора': 0.65446657, 'магазин': 0.65467703, 'удача': 0.65482104, 'волонтер': 0.65544796, 'сексуально': 0.6555309, 'ордер': 0.6557747, 'точка': 0.65612084, 'через': 0.6563236, 'глина': 0.65652716, 'значок': 0.65661323, 'плакат': 0.6568083, 'слух': 0.65709555, 'нога': 0.6572164, 'фотограф': 0.65756184, 'ненависть': 0.6578564, 'пункт': 0.65826315, 'берег': 0.65849876, 'альбом': 0.65849936, 'кролик': 0.6587049, 'масло': 0.6589803, 'бензин': 0.6590406, 'покупка': 0.65911734, 'параграф': 0.6596477, 'вакцина': 0.6603271, 'континент': 0.6609991, 'расизм': 0.6614046, 'правило': 0.661452, 'симптом': 0.661881, 'романтика': 0.6626457, 'атрибут': 0.66298646, 'олень': 0.66298693, 'кафе': 0.6635062, 'слово': 0.6636568, 'машина': 0.66397023, 'джаз': 0.663977, 'пиво': 0.6649644, 'слуга': 0.665489, 'температура': 0.66552, 'море': 0.666358, 'чувак': 0.6663854, 'комфорт': 0.66651237, 'театр': 0.66665906, 'ключ': 0.6670032, 'храм': 0.6673037, 'золото': 0.6678767, 'робот': 0.66861665, 'джентльмен': 0.66861814, 'рейтинг': 0.6686267, 'талант': 0.66881114, 'флот': 0.6701237, 'бонус': 0.67013747, 'величина': 0.67042017, 'конкурент': 0.6704642, 'конкурс': 0.6709986, 'доступ': 0.6712131, 'жанр': 0.67121863, 'пакет': 0.67209935, 'твердо': 0.6724718, 'клуб': 0.6724739, 'координатор': 0.6727365, 'глобус': 0.67277336, 'карта': 0.6731522, 'зима': 0.67379165, 'вино': 0.6737963, 'туалет': 0.6744124, 'середина': 0.6748006, 'тротуар': 0.67507124, 'законопроект': 0.6753582, 'земля': 0.6756074, 'контейнер': 0.6759613, 'посольство': 0.67680794, 'солдат': 0.6771952, 'канал': 0.677311, 'норма': 0.67757475, 'штраф': 0.67796284, 'маркетинг': 0.67837185, 'приз': 0.6790007, 'дилер': 0.6801595, 'молитва': 0.6806114, 'зона': 0.6806243, 'пояс': 0.6807122, 'автор': 0.68088144, 'рабство': 0.6815858, 'коридор': 0.68208706, 'пропаганда': 0.6826943, 'журнал': 0.6828874, 'портрет': 0.68304217, 'фермер': 0.6831401, 'порошок': 0.6831531, 'сюрприз': 0.68327177, 'камера': 0.6840434, 'фаза': 0.6842661, 'природа': 0.6843757, 'лимон': 0.68452585, 'гараж': 0.68465877, 'рецепт': 0.6848821, 'свинина': 0.6863143, 'атмосфера': 0.6865022, 'режим': 0.6870908, 'характеристика': 0.6878463, 'спонсор': 0.6879278, 'товар': 0.6880773, 'контакт': 0.6888988, 'актриса': 0.6891222, 'диск': 0.68916976, 'шоколад': 0.6892894, 'банда': 0.68934155, 'панель': 0.68947715, 'запуск': 0.6899455, 'травма': 0.690045, 'телефон': 0.69024855, 'список': 0.69054323, 'кредит': 0.69054526, 'актив': 0.69087565, 'партнерство': 0.6909646, 'спорт': 0.6914842, 'маршрут': 0.6915196, 'репортер': 0.6920864, 'сегмент': 0.6920909, 'бунт': 0.69279015, 'риторика': 0.69331145, 'школа': 0.6933826, 'оператор': 0.69384277, 'ветеран': 0.6941337, 'членство': 0.69435036, 'схема': 0.69441277, 'манера': 0.69451445, 'командир': 0.69467854, 'формат': 0.69501007, 'сцена': 0.69557995, 'секрет': 0.6961215, 'курс': 0.6964162, 'компонент': 0.69664925, 'патруль': 0.69678336, 'конверт': 0.6968681, 'символ': 0.6973544, 'насос': 0.6974678, 'океан': 0.69814134, 'критик': 0.6988366, 'доброта': 0.6989736, 'абсолютно': 0.6992678, 'акцент': 0.6998319, 'ремонт': 0.70108724, 'мама': 0.7022723, 'тихо': 0.70254886, 'правда': 0.7040037, 'транспорт': 0.704239, 'книга': 0.7051158, 'вода': 0.7064695, 'кухня': 0.7070433, 'костюм': 0.7073295, 'дикий': 0.70741034, 'прокурор': 0.70768344, 'консультант': 0.707697, 'квартира': 0.7078515, 'шанс': 0.70874536, 'сила': 0.70880103, 'хаос': 0.7089504, 'дебют': 0.7092187, 'завтра': 0.7092679, 'горизонт': 0.7093906, 'модель': 0.7097884, 'запах': 0.710207, 'сама': 0.71082854, 'весна': 0.7109366, 'орган': 0.7114152, 'далекий': 0.7118393, 'смерть': 0.71213734, 'медсестра': 0.71224624, 'молоко': 0.7123647, 'союз': 0.71299064, 'звук': 0.71361446, 'метод': 0.7138604, 'корпус': 0.7141677, 'приятель': 0.71538115, 'центр': 0.716277, 'максимум': 0.7162813, 'страх': 0.7166886, 'велосипед': 0.7168154, 'контроль': 0.7171681, 'ритуал': 0.71721196, 'команда': 0.7175366, 'молоток': 0.71759546, 'цикл': 0.71968937, 'жертва': 0.7198437, 'статус': 0.7203152, 'пульс': 0.7206338, 'тренер': 0.72116625, 'сектор': 0.7221448, 'музей': 0.72323525, 'сфера': 0.7245963, 'пейзаж': 0.7246053, 'вниз': 0.72528857, 'редактор': 0.7254647, 'тема': 0.7256167, 'агент': 0.7256874, 'дизайнер': 0.72618955, 'деталь': 0.72680634, 'банк': 0.7270782, 'союзник': 0.72750694, 'жест': 0.7279984, 'наставник': 0.7282404, 'тактика': 0.72968495, 'спектр': 0.7299538, 'проект': 0.7302779, 'художник': 0.7304505, 'далеко': 0.7306006, 'ресурс': 0.73075294, 'половина': 0.7318293, 'явно': 0.7323554, 'день': 0.7337892, 'юрист': 0.73461473, 'широко': 0.73490566, 'закон': 0.7372453, 'психолог': 0.7373602, 'сигарета': 0.73835427, 'проблема': 0.7388488, 'аргумент': 0.7389784, 'старший': 0.7395191, 'продукт': 0.7395814, 'ритм': 0.7406945, 'широкий': 0.7409786, 'голос': 0.7423325, 'урок': 0.74272805, 'масштаб': 0.74474066, 'критика': 0.74535364, 'правильно': 0.74695253, 'авторитет': 0.74697924, 'активно': 0.74720675, 'причина': 0.7479735, 'сестра': 0.74925977, 'сигнал': 0.749686, 'алкоголь': 0.7517742, 'регулярно': 0.7521055, 'мотив': 0.7527843, 'бюджет': 0.7531772, 'плоский': 0.754082, 'посол': 0.75505507, 'скандал': 0.75518423, 'дизайн': 0.75567746, 'персонал': 0.7561288, 'адвокат': 0.7561835, 'принцип': 0.75786924, 'фонд': 0.7583069, 'структура': 0.75888604, 'дискурс': 0.7596848, 'вперед': 0.76067656, 'контур': 0.7607424, 'спортсмен': 0.7616756, 'стимул': 0.7622434, 'партнер': 0.76245433, 'стиль': 0.76301545, 'сильно': 0.7661394, 'текст': 0.7662303, 'фактор': 0.76729685, 'герой': 0.7697237, 'предмет': 0.775718, 'часто': 0.7780384, 'план': 0.77855974, 'рано': 0.78059715, 'факт': 0.782439, 'конкретно': 0.78783923, 'сорок': 0.79080343, 'аспект': 0.79219675, 'контекст': 0.7926827, 'роль': 0.796745, 'президент': 0.8007479, 'результат': 0.80227, 'десять': 0.8071967, 'скоро': 0.80976427, 'тонкий': 0.8100516, 'момент': 0.8120169, 'нести': 0.81280494, 'документ': 0.8216758, 'просто': 0.8222313, 'очевидно': 0.8242744, 'точно': 0.83183587, 'один': 0.83644223, 'пройти': 0.84026355}

ways to improve:

• remove potential bad words from training set

• expand looking for candidate words by doing predictable changes a la <_(@Sira2019) “Towards an automatic recognition of mixed languages: The Ukrainian-Russian hybrid language Surzhyk” (2019) / Nataliya Sira, Giorgio Maria Di Nunzio, Viviana Nosilia: z / http://arxiv.org/abs/1912.08582 / _>

• add weighting based on frequency, rarer words will have less stable embeddings

• look at other trained vectors, ideally sth more processed

• And actually thinking about it — is there anything I can solve through this that I can’t solve by parsing one or more dictionaries, maybe even making embeddings of the definitions of the various words?

  • That said most other research on the topic of automatically finding cognates had this issue as well
  • And no one did this that way, and no one ever did this for RU/UA

Fazit: leaving this alone till after the masterarbeit as a side project. It’s incredibly interesting but probably not directly practical. Sad.

Jupyter-lab

By default, <Esc> — bad idea for the same reason in vim it’s a bad idea.

AND my xkeymap-level keyboard mapping for Esc doesn’t seem to work here.

Default-2 is <C-]> which is impossible because of my custom keyboard layout.

Will be <C-=>.

        {
            "command": "vim:leave-insert-mode",
            "selector": ".jp-NotebookPanel[data-jp-vim-mode='true'] .jp-Notebook.jp-mod-editMode",
            "keys": [
                "Ctrl =",
            ]
        }

(I can’t figure out why ,l etc. don’t work in jupyterlab for this purpose)

vim

(<leader> is ,)

"Insert mode mappings
" Leave insert mode
imap <leader>l <Esc>
imap qj <Esc>

" Write, write and close
imap ,, <Esc>:x<CR>
map ,. :w<CR>

… I will have an unified set of bindings for this someday, I promise.

• Sources first, text second: I spent a lot of time polishing specific sentences etc., loosely linking relevant sources, but when needing to actually add sources finding a specific one to support a statement is hard, even if I have multiple ones supporting different parts of the statement or the spirit of it
  • 
    • some people translate book names into English on Wikipedia, sourcing the Ukrainian one and the translation is right and they are awesome etc., but that was unexpected, hah
    • and generally dealing with translations is hard
• English-language sources for common Ukrainian bits is hard
  • the executed renaissance - two papers about it, both critical of it, but finding something that confirms that it’s a thing that exists is not trivial
    • pro-ER: article on a website The Executed Renaissance: The Book that Saved Ukrainian Literature from Soviet Oblivion | Article | Culture.pl
    • anti-ER: the only two published citeable sources I ound
      • Arguments against ‘The Executed Renaissance’
      • The Executed Renaissance Paradigm Revisited on JSTOR
• Generally the approach should be as mentioned by I think Scott Alexander or someone in 231002-2311 Meta about writing a Masterarbeit: follow the sources wherever they lead, instead of writing something when inspired and then look for sources. You may be right but much less rephrasing if you do it starting from sources to begin with.
• Focus and prioritize
  • the approach I do for tasks/theory/steps should have been done for the text as well. Ukrainian language history is the least relevant part, even if interesting to me.
• Added the ‘use pycharm earlier bit’ to 231207-1937 Note to self about OOP
• Spend less time on Ukrainian grammar and more on other eval harnesses and the literature, incl. other cool packages that exist

Copypasting this (still draft version) here in full, before radically shortening it for my master thesis.

Historical context and bilingualism in the modern Ukrainian language

L’Ukraine a toujours aspiré à être libre
“Ukraine has always aspired to be free.” Voltaire, 1731 ¹

This section describes the bilingual nature of Ukraine’s society and the impact of historical state policies on the modern development of the language.

The ongoing Russian invasion is viewed by many as a continuation of a long-standing historical pattern, rather than an isolated incident.

This section doesn’t attempt to justify or challenge any particular position regarding the events described, nor is meant to be a definitive account of the history of the language.

But I believe this perspective is important to understanding the current linguistic landscape in Ukraine, as well as the linguistic challenges and phenomena that had a direct relevance on this thesis. (TODO mention how and which tasks are impacted by this)

A historical overview

• todo: more synonyms for ‘policy’
• todo: better title
• todo: sources for everything-everything-everything
• sources
  • 1987 book about the entire topic [^@krawchenko1987social]
  • Article The Executed Renaissance: The Book that Saved Ukrainian Literature from Soviet Oblivion | Article | Culture.pl
  • Keeping a record is the best book on this [^@1130282272476965120]

In Ukraine itself, the status of Ukrainian (its only official language) varies widely, but for a large part of Ukrainians the question was never too much on the foreground (until recently, that is).

A significant number of people in Ukraine are bilingual (Ukrainian and Russian languages), and almost everyone can understand both Russian and Ukrainian.²
The reasons for this include Ukraine’s geographical and cultural proximity to Russia, and was to a large extent a result of consistent policy first of the Russian empire and the Soviet Union.

The suppression of Ukrainian in the Russian Empire

In the Russian Empire, the broader imperial ideology sought to assimilate various ethnicities into a single Russian identity (with Russian as dominant language), and policies aimed at diminshing Ukrainian national self-consciousness were a facet of that³. TODO source

Ukrainian (then officially called little Russian language/малорусский язык) was stigmatized as a (uncultured town folks’) dialect of Russian, unsuited for ‘serious’ literature or poetry — as opposed to the great Russian language (not editorializing, it was literally called that; these phrasing applied to the names of ethnicities as well, Russia as great Russia and Ukraine as little Russia; the extent to which this referred broader cultural attitudes is a discussion out of scope of this Thesis). (TODO footnote to ‘War and Punishment’ for more on this)

The history of Ukrainian language bans is long enough to merit a Wikipedia page itemizing all the attempts, ⁴ with the more notable ones in the Russian Empire being the 1863 Valuev Circular (forbidding the use of Ukrainian in religious and educational printed literature) and the Ems Ukaz, a decree by Emperor Alexander II banning the use of the Ukrainian language in print (except for reprinting old documents), forbidding the import of Ukrainian publications and the staging of plays or lectures in Ukrainian (1876). (TODO sources for both)

The redefinition of Ukrainian in the Soviet Union

• TODO [^@marshall2002post] has many sources for this! The first decade of Soviet Union brought Ukrainisation as part of a new Soviet nationalities policy, and the use of Ukrainian in different areas was promoted. The reason for this was chiefly ideological — the ghosts of the old empire were considered a danger and encouraging formerly-suppressed cultures was one way to fight them — but nevertheless, this period led to a short-lived period of flourishing for Ukrainian literature and culture in general.

The 1928 grammar reform (sometimes called Skrypnykivka after the minister of education Skrypnyk) passed during this period, drafted by a commitee of prominent Ukrainian linguists, writers, and teachers synthetized the different dialects into a single orthography to be used across the entire territory.

The Ukrainian writers and intellectuals of that period became known as “the executed Renaissance”: most of them were purged in the years to follow, after the Soviet Union took a sharp turn towards Russification in the late 1920s and in the multiple waves of purges that followed. (Most prominent members of committee behind Skrypnykivka were repressed as well; Skrypnyk himself committed suicide in 1933.)

A new ‘orthographic’ reform was drafted in 1933. It had the stated goal of removing alleged burgeoise influences of the previous one. Andriy Khvylia⁵, the chairman of the new Orthography Commission described in his 1933 book “Eradicate, Destroy the Roots of Ukrainian Nationalism on the Linguistic Front” (TODO source) how the new reform eliminates all “deadly conservative norms established by nationalists” that “focused the Ukrainian language on the Polish and Czech borgeois cultures (…) and set a barrier between the Ukrainian and Russian language”.

In practice the reform brought the Ukrainian language much closer to Russian in many ways:

• Grammatically, by introducing relatively major changes in declension paradigms, plurals, especially targeting grammatical structures absent in Russian. The letter ґ (absent in the Russian alphabet) was dropped.
• Changes in the vocabulary were also massive: the spelling of foreign-derived words and proper names were changed to fit Russian patterns. For example, the German diphthong ei (IPA: [aɪ], approx. English “eye”), in Ukrainian ай with the same sound, became the Russian ей (IPA: [ɛj], approx. in English “they”), changing surnames like Einstein / Айнштайн (IPA: [ˈajnʃtajn])[^44] to Ейнштейн (IPA: [ˈɛjnʃtɛjn]). Genders of words were changed.
• Terminology: there was an effort to eliminate Ukrainian-specific vocabulary, much of which was just replaced with the (sometimes completely different) Russian equivalents. This was quite explicit, to the point that publishing houses regularly received lists of words to avoid. This increased the dependence of Ukrainian on Russian science.

Many Ukrainian writers, poets and dissidents kept using the ‘old’ orthography, as well as the Ukrainian community outside the Soviet Union.

After the fall of the Soviet Union, there were many proposals for restoring the original orthography, but only the letter ґ was restored. In 2019 a new version of the Ukrainian orthography was approved, which restored some of the original rules as ’legal’ variants but without mandating any of them.

EDIT: this is becoming a more generic thingy for everything I’d ever need to refer to when writing a paper, later I’ll clean this mess.

Resources – DREAM Lab links to https://dream.cs.umass.edu/wp-content/uploads/2020/04/Tips-and-Best-Practices.pdf. Until I set up a system to save PDF info, I’ll paste it as screenshots here:

ChatGPT summarized the relevant pages of the PDF file thus, but didn’t do it well, mostly rewriting myself:

General Formatting

• Smart Quotes: Use `x’ and ``x’’ for single and double quotes respectively.
• Large Numbers: Write large numbers as 54{,}000.
• Word Breaking: Use - to suggest hyphenation points in long words, including ones that already have hyphens inside them. Such words are likely to go over margins etc.: multi-discipli\-nary
  • EDIT: If I don’t want a word to get broken, use mbox: \caption{Worldwide distribution of \mbox{plantNaturalist-}500k}¹
  • EDIT2: the extdash package is even better, same link
  • EDIT3: If you want to allow linebreaks somewhere but without a dash and only if needed, it’s long\allowbreak word²
• Overflow Lines: \begin{sloppypar}... for paragraphs where latex goes over the margin.
• Float Positioning: Place figures and tables at the top of the page for readability: \begin{figure}[t]
• Center Align Floats: Use \centering for aligning tables and figures.

Specific Elements

• Non-Breaking Space: Use the tilde (~) to keep words together, including always
  • before citations: sth~\cite{whatever}
  • after numbers in text
  • before figure/section references
• Emphasize vs. Bold: Prefer \emph over bold or \textit.

Equations and Macros

• Macros: Create macros for things that will be changed later, such as DATASET NAMES. \newcommand{\system}{SQuID\xspace}
  • \xspace here adds as space unless end of sentence. Package \usepackage{xspace}

Spacing and Sizing

• Paragraph Spacing: Use \smallskip, \medskip, and \bigskip, instead of \vspace
• Line and Text Width: Use fractions of \linewidth or \textwidth.
• Resizing Tables/Figures: Use \resizebox with appropriate dimensions.

Other bits I’ll need but don’t want to transcribe but need to mention at least once so I can search for them later

Compression hacks (see pics)

Paper writing hacks:

Even more

best practices - When should I use non-breaking space? - TeX - LaTeX Stack Exchange lists ALL the places where Knuth wanted people to put nonbreaking spaces, incl:

• 1)~one 2)~two
• Donald~E. Knuth
• 1,~2
• Chapter~12

Less obvious and not from him:

• I~am

Also:

• before all cites, refs
• … and inlined equations

ALSO

ChatGPT says that citations should come before footnotes to prioritize the scholarly source over unimportant info. So this [32] ³ and not this³ [32]. Basically footnotes after all punctuation and citations. OK

EDIT: NOT PARENTHESES, THEY SHOULD BE WITHIN PARENTHESES.⁴ DAMN

ALSO

I sometimes write and around ~50% forgetting that ~ is a nbsp — hard to catch when reading the text.

ALSO

As when writing code I like to add some assert False (or a failing test) so that I know where I stopped the last time, \latexstopcompiling here is a neat way to make sure I REALLY finis ha certain line I started but not finished.

Rounding.

Previously: 211018-1510 Python rounding behaviour with TL;DR that python uses banker’s rounding, with .5th rounding towards the even number.

• Rounding - Wikipedia

• Nearest Integer Function – from Wolfram MathWorld

• Floor/ceil have their usual latex notation as \rceil, \rfloor (see LaTeX/Mathematics - Wikibooks, open books for an open world at ‘delimiters’)

• “Normal” rounding (towards nearest integer) has no standard notation: ceiling and floor functions - What is the mathematical notation for rounding a given number to the nearest integer? - Mathematics Stack Exchange

  • $\lfloor3.54\rceil$ is one notation people mention
  • Some suggest $[3.54]$ but most hate this because overused
  • $nint(3.45)$ (’nearest integer function’) works
  • let XXX denote the standard rounding function

• Bankers’ rounding (that python and everyone else use for tie-breaking for normal rounding and .5) has no standard notation as well

  • Let $\lfloor x \rceil$ denote "round half to even" rounding (a.k.a. "Banker's rounding"), consistent with Python's built-in round() and NumPy's np.round() functions.

Require/ensure

Require/Ensure is basically Input/Output and can be renamed thus¹:

\floatname{algorithm}{Procedure}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

Sample algpseudocode algorithm float

\usepackage{algorithm}
\usepackage{algpseudocode}

% ...

\begin{algorithm}
\caption{Drop Rare Species per Country}
\label{alg:drop}
\begin{algorithmic}
\Require $D_0$: initial set of occurrences
\Ensure $D_1$: Set of occurrences after filtering rare species
\State $D_1 \gets$ \emptyset
\For{each $c$ in Countries}
    \For{each $s$ in Species}
        \If {$|O_{c,s} \in D_0| \geq 10$} % if observations of species in country in D_0 have more than 10 entries; || is set cardinality
            \State{$D_1 \gets D_1 \cup O_{c,s}$} 
        \EndIf
    \EndFor
\EndFor
\end{algorithmic}
\end{algorithm}

List of all map providers, not all included in geopandas and some paid, nevertheless really neat: https://xyzservices.readthedocs.io/en/stable/gallery.html

For the 231024-1704 Master thesis task CBT task of my 230928-1745 Masterarbeit draft, I’d like to create an ontology I can use to “seed” LMs to generate ungoogleable stories.

And it’s gonna be fascinating.

I don’t know what’s the difference between knowledge graph, ontology etc. at this point.

Basics

I want it to be highly abstract - I don’t care if it’s a forest, if it’s Cinderella etc., I want the relationships.

Let’s try. Cinderella is basically “Rags to riches”, so:

• Character roles
  • Protagonist
    • Underprivileged protagonist (Cinderella)
  • Benefactor
  • …
• Key plot points
  • Hardship
  • Opportunity
  • Transformation
  • Achievement
• Thematic elements
  • Transformation
  • Reward?..

…

Or GPT3’s ideas from before:

"Entities": {
	"Thief": {"Characteristics": ["Cunning", "Resourceful"], "Role": "Protagonist"},
	"Fish": {"Characteristics": ["Valuable", "Symbolic"], "Role": "Object"},
	"Owner": {"Characteristics": ["Victimized", "Unaware"], "Role": "Antagonist"}
},
"Goals": {
	"Thief": "Steal Fish",
	"Owner": "Protect Property"
},
"Challenges": {
	"Thief": "Avoid Detection",
	"Owner": "Secure Property"
},
"Interactions": {
	("Thief", "Fish"): "Theft",
	("Thief", "Owner"): "Avoidance",
	("Owner", "Fish"): "Ownership"
},
"Outcomes": {
	"Immediate": "Successful Theft",
	"Long-term": "Loss of Trust"
},
"Moral Lessons": {
	"Actions Have Consequences",
	"Importance of Trust",
	"Greed Leads to Loss"
}

• ENTITIES have a:
  • ROLE
    • Protagonist
  • CHARACTERISTICS
    • Poor

Awesome chats with ChatGPT

Here’s it generating an ontology based on the above graph: https://chat.openai.com/share/92ed18ce-88f9-4262-9dd9-f06a07d06acc

And more in UKR: https://chat.openai.com/share/846a5e85-353e-4bb5-adbe-6da7825c51ed

Tale ontology v0.0.1

In bold bits I’m not sure of. In decreasing order of abstraction, with the first two being the most generic ones and the latter ones more fitting for concrete stories.

• KEY PLOT POINTS:
  • Hardship
  • Disruption
  • Opportunity
  • Transformation
  • Achievement
• CHARACTER ROLES:
  • Description: most abstract roles for the characters
  • Values:
    • Protagonist
    • Antagonist
    • Object
    • Benefactor
    • Goal object
    • Ally
    • etc.
• ENTITIES
  • Description:
    • can be characters, important objects, whatever
  • Attributes:
    • Characteristics:
      • … basically adjectives.
      • e.g. Resourceful, motivated, hungry
    • Role: CHARACTER ROLE
• GOALS
  • Description: what drives the character/entity
  • Attributes:
    • Entity: ENTITY
    • Goal: main goal of entity in this context
  • Example:
    • Thief: steal fish
    • Owner: protect property
• INTERACTIONS
  • Plaintext description of what’s up between pairs of ENTITIES
  • Example: (’thief’,‘fish’): theft
• OUTCOMES
  • Attributes:
    • SHORT-TERM: plaintext description
    • LONG-TERM: plaintext description
  • Example: “loss of trust”
  • Optional
• MORAL LESSONS:
  • Description: short plaintext description
  • Optional
  • examples:
    • “Actions Have Consequences”, “Importance of Trust”, “Greed Leads to Loss”

Remaining issues:

• Can I generalize to remove “fish” and stuff? Maybe call it object?
• Two versions of the ontology with different levels of abstractions?

Test 1:

Here’s ChatGPT applying that to Shrek: https://chat.openai.com/share/d96d4be6-d42f-4096-a18f-03f786b802c6

Modifying its answers:

“Using this ontology for abstract fairy tale description, please create a generalized graph structure for THE FIRST HARRY POTTER MOVIE. Focus on the overarching themes and character roles without specific names or unique settings. The graph should include key plot points, character roles, entities, goals, interactions, outcomes, and moral lessons, all described in a manner that is broadly applicable to similar stories.”

Revised Ontology for Abstract Fairy Tale Description

Character Roles

• Protagonist: Central character facing a challenge.
• Antagonist: Character or force opposing the protagonist.
• Goal Object: The objective or person the protagonist seeks.
• Ally: A supportive character aiding the protagonist.

Entities

• Description: Characters or significant elements in the story.
• Attributes:
  • Characteristics: Abstract adjectives describing the entity.
  • Role: One of the defined character roles.

Goals

• Description: Motivations driving the entities.
• Attributes:
  • Entity: Reference to an entity.
  • Objective: Abstract goal of the entity.

Interactions

• Description: General interactions between pairs of entities.
• Format: (‘Entity1’, ‘Entity2’): Type of interaction.

Outcomes

• Short-term: Immediate result of the story’s climax.
• Long-term: Lasting changes or lessons from the story.

Moral Lessons

• Description: Universal themes or morals derived from the story.

Attempt 2

<

Tools

• Tutorial: Build a Knowledge Graph using NLP and Ontologies - Developer Guides