Feature Request: On-the-fly tokenization for dataloader
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Is your feature request related to a problem? Please describe.**
The biggest pain point with Megatron-LM's dataloader is the requirement to tokenize data offline before training.
This creates significant friction:
• Retokenization can take several hours
• Experimentation is painful when testing different chat templates
• Post-training workflows suffers too - SFT/RLHF need quick iteration on templates
**Describe the solution you'd like**
Support for on-the-fly tokenization that accepts raw JSON/JSONL data directly, eliminating the offline preprocessing step. This could be implemented as:
- A new dataset class (e.g., `GPTRawTextDataset`) that reads text and tokenizes during data loading, OR
- A new low-level dataset that plugs into the existing `GPTDataset`
**Describe alternatives you've considered**
Megatron Energon supports it but it's only for WebDataset and multimodal data
**Additional context**
CC @Skylion007
Contributor guide
Assessment
This issue has not been assessed yet.