facebookresearch / facebookresearch/RAM
Bug report in otb project
- Dominant language
- Python
- Stars
- 382
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
In UnderThinkingBench, there exists some items without "source_dataset", but "source" instead with values "aime" and "hmmt", which seems to be in the UnderThinkingBench-Math subset.
UnderThinkingBench entry:
https://github.com/facebookresearch/RAM/blob/264c0479aa2df0ea697a0fadbb10356d1e4ba41f/projects/otb/eval.py#L39-L40
Error encountered here:
https://github.com/facebookresearch/RAM/blob/264c0479aa2df0ea697a0fadbb10356d1e4ba41f/projects/otb/evals/underthink_eval.py#L33-L34
Possible solution in `eval.py`:
```python
import json
# ...
elif subset == "underthinking-bench" or "underthinking" in subset:
metadata = json.loads(row["metadata"])
if "source" in metadata and metadata["source"] in ["aime", "hmmt"]:
acc = eval_math(row, tokenizer, model_name)
else:
acc = eval_underthink(row)
```
Contributor guide
Assessment
This issue has not been assessed yet.