In the middle of the desert you can say anything you want

29 Jan 2024


From the eleutherAI eval harness (240129-1833 Writing evaluation code for my Masterarbeit) I learned about promptsource templates for various tasks!

Basically crowdsourced(?) templates for LLMs for specific well-known tasks like

answer_choices: Guaranteed ||| Possible ||| Impossible
jinja: "Assume it is true that {{premise}} \n\nTherefore, \"{{hypothesis}}\" is\
\ {{\"guaranteed\"}}, {{\"possible\"}}, or {{\"impossible\"}}? ||| {{ answer_choices[label]\
\ }}"
Nel mezzo del deserto posso dire tutto quello che voglio.