huggingface / huggingface/model-evaluator
Extend zero-shot classification language task to support a generalized likelihood task
- Dominant language
- Python
- Stars
- 14
- Forks
- 8
- Avg merge
- 34m
- Merged PRs (30d)
- 1
Description
Add a follow-up version of the zero-shot classification task where instead of comparing endings like concat(PROMPT, ENDING1) vs. concat(PROMPT, ENDING2) it's a generalized likelihood task where you specify a prompt w an arbitrary number of placeholders like so:
`the woman walked into a {MASK} and started {MASK} afterward.`
and there's a column which takes a list of substitutions in order (as a list of lists, where each sublist contains a set of substitutions), e.g.
`[["cafe", "to order a drink"]], ["forest", "frolicking"], ["tree", "hurting"]`
To compare the likelihoods of:
`the woman walked into a cafe and started to order a drink afterward`
`the woman walked into a forest and started frolicking afterward`
...
This extension would enable a lot more tasks to be done zero-shot (e.g. any coreference task, winograd schema type tasks).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.