hexhowells / hexhowells/tinyGPT
Wrong max_iters in config
Open
Beginner friendly
Nobody has claimed this yet.
bug
- Dominant language
- Jupyter Notebook
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The max_iters parameter in config.toml is set to 50,000. However, this model is trained with a batch size of 512, sequence length of 1024, and is trained on fineweb-10BT which contains 10 billion tokens. Therefore:
iters = \frac{10,000,000,000}{512 \times 1024} \approx 19,000
Thus the max_iters value should be around 19,000 instead.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Open config.toml and inspect max_iters alongside the stated batch size, sequence length, and FineWeb-10BT token count. Update the value to approximately 19,000, then verify that the configuration reflects the intended training calculation.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100