size not match error after many training steps
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
After many training steps, the below error often occurs:
RuntimeError: The size of tensor a (6236) must match the size of tensor b (8192) at non-singleton dimension 0
I even add the below code to ensure their size is the same:
student_log_probs = [s_log_prob[-response_length:] for s_log_prob, response_length in zip(student_log_probs, response_lengths)]
so what are the possible reasons? Has anyone else encountered the same problem?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at slime/backends/megatron_utils/loss.py around line 296 and inspect how student_log_probs and response_lengths are produced before the failing tensor operation. Reproduce the mismatch after extended training, trace the differing lengths, and document or fix the cause so the tensors remain compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100