NVIDIA / NVIDIA/Megatron-LM

Feature Request: On-the-fly tokenization for dataloader

Open
#2,727 1 comment 1 reaction 0 assignees View on GitHub
enhancement module: data pipeline
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Is your feature request related to a problem? Please describe.**

The biggest pain point with Megatron-LM's dataloader is the requirement to tokenize data offline before training.

This creates significant friction:
• Retokenization can take several hours
• Experimentation is painful when testing different chat templates
• Post-training workflows suffers too - SFT/RLHF need quick iteration on templates

**Describe the solution you'd like**
Support for on-the-fly tokenization that accepts raw JSON/JSONL data directly, eliminating the offline preprocessing step. This could be implemented as:
- A new dataset class (e.g., `GPTRawTextDataset`) that reads text and tokenizes during data loading, OR
- A new low-level dataset that plugs into the existing `GPTDataset`

**Describe alternatives you've considered**
Megatron Energon supports it but it's only for WebDataset and multimodal data

**Additional context**

CC @Skylion007

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.