facebookresearch / facebookresearch/maca
Difference in expected total and actually loaded examples
- Dominant language
- Python
- Stars
- 7
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
I tried running the code for Qwen2B and GSM8K (1500-500) dataset for DPO training. I noted that final reported responses are not for the full dataset and also the code takes it yo account -
* "present" metrics use the actually *loaded* example counts.
* "all" metrics use the provided `expected_total` (missing = incorrect).
Why is there difference in loaded and expected examples for test set and also for training? Why are there failures while processing examples in code?
Contributor guide
Research direction
Reproduce Qwen2B DPO training with the GSM8K 1500-500 dataset. Compare the actually loaded example counts with expected_total for both training and test data, and inspect the processing failures reported by the code. Done means identifying why counts differ and what causes the failed examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100