From the eleutherAI eval harness (240129-1833 Writing evaluation code for my Masterarbeit) I learned about promptsource templates for various tasks!

Basically crowdsourced(?) templates for LLMs for specific well-known tasks like

• they look rather complex and jinja-based promptsource/promptsource/templates/wino_bias/type1_pro/templates.yaml at main · bigscience-workshop/promptsource
• Multiple-choice SNLI: promptsource/promptsource/templates/snli/templates.yaml

answer_choices: Guaranteed ||| Possible ||| Impossible
jinja: "Assume it is true that {{premise}} \n\nTherefore, \"{{hypothesis}}\" is\
\ {{\"guaranteed\"}}, {{\"possible\"}}, or {{\"impossible\"}}? ||| {{ answer_choices[label]\
\ }}"

• Yahoo Answers guess-the-question-topic! promptsource/promptsource/templates/yahoo_answers_topics/templates.yaml at main · bigscience-workshop/promptsource