serhii.net

Tensorflow gradient propagation

• ValueError: No gradients provided for any variable in Tensorflow - Stack Overflow

When you do annotation_pred = tf.to_float(tf.argmax(out, dimension=4, name='prediction')), you get an index of the max value in your tensor. This index can’t be derivated, thus the gradient can’t flow throught this operation.

So as your loss is only defined by this value, and the gradient can’t flow throught it, no gradient can be calculated for your network.

Argmax is okay if I don’t calculate my loss through it.

Python / Numpy ellipsis (…)

• Cookbook/Indexing - SciPy wiki dump

The ellipsis (three dots) indicates “as many ‘:’ as needed” This makes it easy to manipulate only one dimension of an array, letting numpy do array-wise operations over the “unwanted” dimensions. You can only really have one ellipsis in any given indexing expression, or else the expression would be ambiguous about how many ‘:’ should be put in each.

English vocabulary

• glib - superficial, shallow, persuasive but insincere in nature.
• retrodict - Wiktionary - to attempt to estimate the previous state from the present.
  • Also interesting is postdiction - Wiktionary, the construction of past conditions by relying on the present.

German RE / AW

Outlook. What is the meaning of “AW” in an email header? – AW == RE in most other languages

Intellij ideavim toggle case

Added the following to .ideavimrc: map <leader>c :action EditorToggleCase<CR>

Tensorflow

Using ‘categorical_crossentropy’ instead of ‘sparse_categorical_crossentropy’, give weird unintuitive errors

ML / NER / Stack / To read

This is a really nice tutorial with the basics that’s not too basic: Sequence Tagging with Tensorflow

Tensorflow metrics ignored if loss doesn’t get defined

So I don’t forget, Metrics ignored when using model.add_loss() (like in VAE example) · Issue #9459 · keras-team/keras · GitHub currently happens.

Keras

• GitHub - fchollet/keras-resources: Directory of tutorials and open-source code repositories for working with Keras, the Python deep learning library

Tensorflow GloVe embeddings

• TensorFlow 07: Word Embeddings (2) – Loading Pre-trained Vectors – Night Cáfe

Intellij IDEA vim plugin

It supports the following :set commands: ideavim/set-commands.md at master · JetBrains/ideavim · GitHub. Especially relativenumbers is nice.

• switch to split screen in IntelliJ (ideaVim) - Stack Overflow for windows splits – Ctrl + ww for quickly changing between splits.
• :source ~/.ideavimrc works.
• vim - Escape to IntelliJ IDEA shortcuts from IdeaVim - Stack Overflow some neat escaped IdeaVim shortcuts

apt-get purge and zsh

zsh does its own wildcard stuff, and apt-get purge nvidia* doesn’t work because of this. apt-get purge nvidia\* does (or with ‘’s). Same story as with scp, I’m surprised I keep having issues with this.

Linux see history of apt-get

• /var/log/apt/history/ contains the last one and the rotated gzipped old ones.
• How do I show apt-get package management history via command line? - Ask Ubuntu. The gzipped ones can be read by…

Linux z-commands (zcat, zless, zgrep, zdiff)

• Unix Z Commands – Zcat, Zless, Zgrep, Zegrep and Zdiff Examples

Pandas

• python - Pandas split column of lists into multiple columns - Stack Overflow

English Pronunciation

Google has nice animations for this!

• Query = [kwee ree]
• Paradigm = [pa ruh daim] or American [peh ruh daim]

Linear algebra

I’ll be following this: 9.1. Attention Mechanism — Dive into Deep Learning 0.7 documentation

• Inner product is the generalization of the dot product. Result is a scalar. ¹

Python assert statement

UsingAssertionsEffectively - Python Wiki assert condition, message -> if condition is false, it returns an AssertionError.

ML

Most of this while I’m reading the “Attention is all you need” paper. The most important resources will be The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time and 9.3. Transformer — Dive into Deep Learning 0.7 documentation.

Definitions

• BLEU is a metric of how good machine translation is.
• Gentle Introduction to Transduction in Machine Learning

Induction, deriving the function from the given data. Deduction, deriving the values of the given function for points of interest. Transduction, deriving the values of the unknown function for points of interest from the given data. {:height=“500px”}

• Positional encoding in the Transformer is very well described at 9.3. Transformer — Dive into Deep Learning 0.7 documentation, with a visualization. Needed because there is no notion of the order of words in the architecture ¹ We can’t do n=1..10 because sentences have different lengths, and word 3 out of 10 is not the same as 3 out of 3.
  • “The intuition here is that adding these values to the embeddings provides meaningful distances between the embedding vectors once they’re projected into Q/K/V vectors and during dot-product attention” ²
• Subword algorithms are ways to represent words that use elements bigger than characters but lower than a word embedding, for example prefixes and suffixes, to better handle unseen words. Byte-pair and word-piece encodings are used by the Transformer.[^swa]
• In essence, label smoothing will help your model to train around mislabeled data and consequently improve its robustness and performance. ³

[^swa] (3 subword algorithms help to improve your NLP model performance)

Resources

• Attention? Attention! is a really nice intro to Attention in general. And the blog itself (Lil’Log) is also absolutely fascinating.
• neural networks - What exactly are keys, queries, and values in attention mechanisms? - Cross Validated
• The Transformer – Attention is all you need. - Michał Chromiak’s blog + the other 3 posts in the blog.
• Dive into Deep Learning — Dive into Deep Learning 0.7 documentation this look really cool, has all the basics and is executable. Another candidate for main textbook.
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time is absolutely wonderful in all aspects.

Stack

• I should make a better Bash timer that counts till the end of the hour, so I don’t have to do this in my head
• I should make a vim keybinding or script that automagically creates Markdown references. (I’d be surprised if this hasn’t been done)

English

Anaphora: * Repetition of something (rhetoric) * Pronouns and references to something already mentioned to avoid repetition (she, it, etc.)

Unpack Java jars

Apparently java files are archives! Laut Extracting the Contents of a JAR File (The Java™ Tutorials > Deployment > Packaging Programs in JAR Files) it’s jar xf jar-file [archived-file(s) to extract].

Pandas and numpy have different std/stdev standard deviation functions!

Both are correct, one divides by N, the other by N-1.

t df.std()==np.std(df, ddof=1). Somethingsomething delta degrees of freedom. ddof=0 can also be passed to pandas.