modelscope / modelscope/ms-swift

About IterableDateset with dataloader_num_workers>1

Open
#6,758 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

  1. 如果使用IterableDataset, dataloader_num_workers=1会被执行:https://github.com/modelscope/ms-swift/blob/3bb1d21959687137b43d4d730424604dcaec68ee/swift/trainers/mixin.py#L74
  2. 如果保证 IterableDataset.num_shards > world_size * dataloader_num_workers, transformers的IterableDatasets是支持dataloader_num_workers>1的;
  3. 如果保证 IterableDataset.num_shards > world_size * dataloader_num_workers 并设置dataloader_num_workers>1,我用ms-swift在单机多卡下可以正常运行,并且训练速度有提升。
    我注意到ms-swift重写了dataloader相关实现:
    https://github.com/modelscope/ms-swift/blob/3bb1d21959687137b43d4d730424604dcaec68ee/swift/trainers/mixin.py#L1139
    我想知道这些部分均可以兼容多机多卡/单机多卡时IterableDataset使用dataloader_num_workers>1吗?另外,目前ms-swift的实现在dataloader_num_workers>1时是否存在一些其他的不兼容现象?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with swift/trainers/mixin.py at the references around lines 74 and 1139, then compare the IterableDataset sharding assumptions with the Transformers implementation. Run focused single-node and multi-node training checks using dataloader_num_workers greater than 1. Done means the supported configurations and any incompatibilities are documented or covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.