TL;DR Lernziele in Moodle

• TODO: print presentation?..

• Points

  • Multiple choice
  • Add comments to code
  • shapes annotieren

• Training

  • Backpropagation etc. - unlikely directly, but drauf aufbauende Fragen können passieren
  • Dropout, early stopping, - Batch normalization, normalisierung/standartisierung - Wir brauchen kein Taschenrechner
  • Paramater sharing
    • same parameters for different data
    • nutzung gleicher Subnetze für die Verarbeitung
  • Verteilte Repräsentazionenen - Lokal (=one/hot etc.) VS Verteilt

• RNN

  • Unroll etc.
  • Different kinds of RNN (LSTM, GRU etc) - diff types unneeded, but the important thing is that information gets lost over a lot of timesteps
    • Neeed to know about the three Gewichtsmatrizen ($W_h$ t y etc.)
  • BiRNN
  • Vanishing Gadietns - far away timestamps get lost along the way, shorter ones bevorzugt gelernt
  • Modellierungsvarianten
    • Stacked RNN - output of layr $n-1$ goes into layer $n$
      • Higher up -> more general features. In NLP lower ones are grammar, higher ones are semantic.
  • Sequenzen I and II and III
    • Ausgabe von der letzte kann als input für den weiteren.

• Convolutions

  • “Patches” - betrachtung einzeler Bildbereiche
  • TODO - convolutions, color/channels, features, x y tiefe features
  • Shapes etc.
  • Zahl der Features entsplicht Filterzahl

• Image classification

  • Different pictures
    • No: “draw me a ResNet”
    • Yes: the tricks that make them nice
  • ResNet is important - and what/how/why are residual connections

• Image segmentation

  • Input: (x,y,c) where c is color
  • I/O shape der Masks: (x,y,m) where m is the number of masks/classes
  • FCN - unpooling
    • Netze im Encoder teil sind +/- gleich, Deconvolution is more interesting
    • Know about the ideas behind these networks
    • U-Net
      • Trick: copy information from the Eingabeschicht a la ResNet
    • Ideen/tricks sind wichtig weil das ist was wir werden letsendlich nützen um unsere Aufgaben zu lösen
  • Object Segmentation
    • Different Modellierungsmöglichtekten (Sl. 37) for input/output Object Detection
    • Typical pipeline - input -> Regions of Interest -> Feature extraction -> ….
    • Wissen Was wird vorhergesagt:
      • Klasse, Bounding box, optionally - pixelmask by some networks
    • Sl. 39 - Fragen about die verschiedee Teile der “Mask R-CNN - Architecture”
      • Feature map etc.
      • Reihenfolge der vier Schrite
      • “Was musste passieren, so that we understand what’s in these regions of interest to find the object class (classification)”
      • What happens if the pictures have different sizes?..
      • Warped feature vectors
      • What happens if the RoI is not correct and the object is bigger?
      • For coordinates - you for example find the central point, and predict the width/heigh
    • Sl. 40 - region proposal network - then you have two outputs - classes and bounding boxes as two separate layers
    • Candidate Anchor boxes
    • YOLO - ergebnisfilter
    • Questions about solutions to the shared tasks?
    • Kernel sizes, shapes, strides - you get the code of a small resnet and you have to annotate the shapes
      • online calculator
  • Sprachverarbeitung
    • ELMo, Stacked BiLSTMs (different functions of each layer + what do you get at the end - both)
    • Padding of sequences
    • No linguistic background
    • Layers sl. 50?
      • Textclassification, classification head wird auf Backbon gesetze
      • First you lookup embeddings, then you do contextualisation to see what it means
      • Ausgabeschicht ist eine menge von Embeddings, then with a classification head you transform them into a solution

• Sequence tagging

  • Not one Vorhersage pro text, sondern eine pro Token
  • NER will have the most questions since we did it in the shared tasks
  • Two ways to do subtokenization - 1-1-1 VS 1-X-X
  • CRF für Übergangswahsrscheinlichkeiten - know why

• Relation extraction - no question

• Sequence-to-sequences+

  • Beam search decodingh as Kompromiss between greedy search and brute force
    • Mehrere Pfade verfolgen und trotzem berechenbar zu bleiben
  • NER als schwerpunkt
  • Bottlenecks solved by attention:
    • Information bottleneck:
      • The information all lives in a single vector between translations
    • Computation bottleneck
      • We need comprimierung and then unzipping, but backpropagation has to go through all of this
      • Sl.63

• Attention

  • Paralellisierbar
    • Query keys und Values
    • Q usually equal to K
    • Q might be the context of the next word
    • Based on this Q you look for an interesting K
      • grammatisch oder semantisch gut passen
    • “Diese sind die wörter (as embeddings) in die Eingangsprache, diese sind die datei die ich habe ,und ich suche was passendes von hier”
  • Information gets multipliziert by the “importance”
  • Self-attention layer will be a focus
    • Context-sensitive Embeddings
    • QKV we get all from the Eingabesequenz
      • “Keine information aus der Zukunft zu nützen” by stting them to -inf., which through softmax becomes zero.
  • Transformer
    • Don’t memorize the graph, but be able to explain th eindividual parts of it
      • No long text-questions with answer, but a lot of different ones, maybe fill-in-the-blanks
    • Vision transformers: won’t be in the textG
  • Foundation models
    • multi-task (self-) supervied training
    • they have “Generisches Domänenwissen”
    • Tranied on a lot of data, maybe different tasks, usually unsupervised
  • One/few/zero shot learning sl. 75
    • Prompt engineering
  • Belohnung/Reward
    • will be on the test because there’s a shared task for it
    • Untershiede between the other variants
    • Other examples of DQN - DDQN, Dueling DQn etc - needed, because it is needed to solve the shared task too - different thins on the text

Shared task:

• with the code from the example:
  • either anpassen the reward
  • or anpassed the algo
• With vanilla DQN probably nothing will happen
• Have patience, 20 episodes won’t be enough
• Very rarely something happens in less than 200 episodes
• “Viel netto arbeitszeit ist es nicht, in eine fokussierte Session sehr gut machbar”
• Ideally have the code and model committed and he can just press play
• “Anweisung”
• Moodle: txt-datei: link to the gitlabplatz with the eval script where it all is inegrated, if there are multiple - mention which ones we abgebe.
• requirements.txt is needed!