facebookresearch / facebookresearch/RAM

Bug report in otb project

Open
#77 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
382
Forks
49
PR merge metrics
No merged PRs in 30d

Description

In UnderThinkingBench, there exists some items without "source_dataset", but "source" instead with values "aime" and "hmmt", which seems to be in the UnderThinkingBench-Math subset.

UnderThinkingBench entry:
https://github.com/facebookresearch/RAM/blob/264c0479aa2df0ea697a0fadbb10356d1e4ba41f/projects/otb/eval.py#L39-L40

Error encountered here:
https://github.com/facebookresearch/RAM/blob/264c0479aa2df0ea697a0fadbb10356d1e4ba41f/projects/otb/evals/underthink_eval.py#L33-L34

Possible solution in `eval.py`:
```python
import json

# ...

elif subset == "underthinking-bench" or "underthinking" in subset:
metadata = json.loads(row["metadata"])
if "source" in metadata and metadata["source"] in ["aime", "hmmt"]:
acc = eval_math(row, tokenizer, model_name)
else:
acc = eval_underthink(row)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.