Determine need for tokenizer assets in MegatronLM checkpoints
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Parent initiative: #2319
## Context
Megatron-Bridge checkpoints may include a `tokenizer/` directory containing tokenizer assets such as vocab and merges files.
It's unclear whether this should be included in the MegatronLM checkpoints.
## Task
Determine whether the tokenizer assets should be saved in MegatronLM checkpoints, and if so create a new issue with the necessary implementation requirements.
Contributor guide
Research direction
Start by reviewing parent issue #2319 and the checkpoint handling described in the issue. Compare whether Megatron-Bridge checkpoints’ tokenizer/ directory and its vocab or merges files are needed in MegatronLM checkpoints. Done means documenting the decision and, if required, creating a follow-up issue with implementation requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100