deepspeedai / deepspeedai/DeepSpeedExamples
Megatron-DeepSpeed Assertion error: last epoch number of samples exceeded max value.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
I am trying to run the megatron-deepspeed gpt pretrain example with 1 GPU but I get this error. Do I have to change something from the pretrain_gpt.sh file?
assert last_epoch_num_samples < (num_samples_per_epoch + 1),
AssertionError: last epoch number of samples exceeded max value.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Megatron-DeepSpeed GPT pretrain example and inspect pretrain_gpt.sh alongside the reported assertion about last_epoch_num_samples. Reproduce the example with one GPU and determine the conditions that trigger the assertion. Done means the documented example runs without this assertion or clearly identifies the required configuration change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100