facebookresearch / facebookresearch/maca

Difference in expected total and actually loaded examples

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
3
PR merge metrics
No merged PRs in 30d

Description

I tried running the code for Qwen2B and GSM8K (1500-500) dataset for DPO training. I noted that final reported responses are not for the full dataset and also the code takes it yo account -
* "present" metrics use the actually *loaded* example counts.
* "all" metrics use the provided `expected_total` (missing = incorrect).

Why is there difference in loaded and expected examples for test set and also for training? Why are there failures while processing examples in code?

Contributor guide

Open the contributing guide

Research direction

Reproduce Qwen2B DPO training with the GSM8K 1500-500 dataset. Compare the actually loaded example counts with expected_total for both training and test data, and inspect the processing failures reported by the code. Done means identifying why counts differ and what causes the failed examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.