about large data to train
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
https://github.com/THUDM/slime/blob/ab880fc8cf5b644011f53b2e4b4e5c405d174873/scripts/run-qwen3-4B-base-sft.sh#L39
when I try to conduct SFT on large data, the process stuck on data load. There are two main problems:
1.why the prompt-data must be a single json or parquet file? Sometime we need to SFT on a large data which may need to be splited to many parts. The framework should support using a file dir including multiple small parts.
2.when I use a single file about 350G containing 15M samples, the process will stuck on dataload and finally failed.
Overall, I have a need to SFT on about 15M samples and how can I achieve this? I cannot use a file dir containing multiple data parts, neither can use single large dataset? Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at scripts/run-qwen3-4B-base-sft.sh#L39 and trace how the prompt data path is loaded. Reproduce the reported behavior with a multi-part directory and a roughly 350G dataset, then verify that large SFT datasets can load without stalling and that the requested input forms are supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100