serhii.net

In the middle of the desert you can say anything you want

04 Dec 2022

Sparse language models are a thing

The GPT31 paper mentioned that it’s 10x bigger than any previous non-sparse LM.

So - sparse LMs () are LMs with A LOT of params where only a subset is used for each incoming example.2

  1. [2005.14165] Language Models are Few-Shot Learners ↩︎

  2. Two minutes NLP — Switch Transformers and huge sparse language models | by Fabio Chiusano | NLPlanet | Medium ↩︎

Nel mezzo del deserto posso dire tutto quello che voglio.
Nel mezzo del deserto posso dire tutto quello che voglio.