meta-pytorch / meta-pytorch/data
Make `ShardingRoundRobinDispatcherIterDataPipe` a subclass of `_ShardingIterDataPipe`
@kurman is already working on this.
Since Feb 6, 2023.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
(Depends on the land of https://github.com/pytorch/pytorch/pull/94095)
DataPipe provides basic unit for building data loading transformation DAG. A lot of DataPipes are RS agnostic (e.g. arbitrary Python per-row or per-batch transformation), while we have seen a few of them requires more first-class citizen support with ReadingService, such as sharding, as they requires different manipulations depends on characteristics:
- Filtering sharding: essentially replicate the upstream data (e.g. when the upstream data is still just metadata such as file list), and drop (N-1)/N of the data.
- Round-robin sharding: to shard already-read data, replicating the upstream data pipe can be too expensive. In such case, we want to have different execution strategy that round robin the elements to different processes.
As such, we refactor to have _ShardingIterDataPipe as a parent class to indicate this special DataPipe in https://github.com/pytorch/pytorch/pull/94095 . As discussed in https://github.com/pytorch/pytorch/pull/94095#discussion_r1097710501, we found ShardingRoundRobinDispatcherIterDataPipe is a different sharding data pipe that has different execution strategy. We can also make it a subclass of _ShardingIterDataPipe and make related changes.
(More background to be added)
cc @ejguan , @kurman
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.