bigscience-workshop / bigscience-workshop/t-zero
"Rank classification" in evaluation for multiple choices
- Dominant language
- Python
- Stars
- 463
- Forks
- 53
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
Thanks for the repo! I was wondering if you would please point out which lines of code are for the "rank classification" idea used for evaluating the multiple-choice style tasks?
The paper describes it like this on Page 6:
> For tasks that involve choosing the correct completion from several options (e.g. multiple choice
question answering), we follow Brown et al. (2020) and use **rank classification** to evaluate our
model: we compute the log-likelihood of each of the target options under the fine-tuned model and
select the option with the highest log-likelihood as the prediction. For simplicity, we do not apply
length normalization to the log-likelihoods of the target options.
Thank you!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.