EleutherAI / EleutherAI/elk-generalization
Support logging of accuracy for non-constant answer choices
Open
- Dominant language
- Python
- Stars
- 34
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Huggingface evaluation doesn't allow passing of the answer choices to the evaluation metric function, so we are currently asserting that the answer choices for all examples are the same.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.