In the middle of the desert you can say anything you want

04 Dec 2022

Sparse language models are a thing

The GPT31 paper mentioned that it’s 10x bigger than any previous non-sparse LM.

So - sparse LMs () are LMs with A LOT of params where only a subset is used for each incoming example.2

Nel mezzo del deserto posso dire tutto quello che voglio.