Compatibility with transformers.Trainer
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
🚀 Feature Request
Currently StreamingDataset handles distributed data parallel training by itself. This makes it incompatible with Trainers that handles data distribution, such as transformers.Trainer (which also distribute the data to the corresponding process based on WORLD_SIZE and RANK).
The request is to add a single node mode to StreamingDataset so that it can also work with trainers that supports dp/ddp.
Motivation
- Many people use Huggingface stack including acclerate, transformers for fine-tuning, especially PEFT.
- People like me love to stick with one data format for dataset storage across multiple training stacks (LLM Foundry and HuggingFace).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how StreamingDataset currently handles distributed data parallel training, then compare that behavior with how transformers.Trainer distributes data using WORLD_SIZE and RANK. Done means a single-node mode lets StreamingDataset work with trainers that handle data distribution, without conflicting process-level sharding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100