modelscope / modelscope/ms-swift
About IterableDateset with dataloader_num_workers>1
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
- 如果使用IterableDataset, dataloader_num_workers=1会被执行:https://github.com/modelscope/ms-swift/blob/3bb1d21959687137b43d4d730424604dcaec68ee/swift/trainers/mixin.py#L74
- 如果保证 IterableDataset.num_shards > world_size * dataloader_num_workers, transformers的IterableDatasets是支持dataloader_num_workers>1的;
- 如果保证 IterableDataset.num_shards > world_size * dataloader_num_workers 并设置dataloader_num_workers>1,我用ms-swift在单机多卡下可以正常运行,并且训练速度有提升。
我注意到ms-swift重写了dataloader相关实现:
https://github.com/modelscope/ms-swift/blob/3bb1d21959687137b43d4d730424604dcaec68ee/swift/trainers/mixin.py#L1139
我想知道这些部分均可以兼容多机多卡/单机多卡时IterableDataset使用dataloader_num_workers>1吗?另外,目前ms-swift的实现在dataloader_num_workers>1时是否存在一些其他的不兼容现象?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with swift/trainers/mixin.py at the references around lines 74 and 1139, then compare the IterableDataset sharding assumptions with the Transformers implementation. Run focused single-node and multi-node training checks using dataloader_num_workers greater than 1. Done means the supported configurations and any incompatibilities are documented or covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100