meta-pytorch / meta-pytorch/data

Make `ShardingRoundRobinDispatcherIterDataPipe` a subclass of `_ShardingIterDataPipe`

Open
#989 0 comments 2 reactions 1 assignee View on GitHub

@kurman is already working on this.

Since Feb 6, 2023.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

(Depends on the land of https://github.com/pytorch/pytorch/pull/94095)

DataPipe provides basic unit for building data loading transformation DAG. A lot of DataPipes are RS agnostic (e.g. arbitrary Python per-row or per-batch transformation), while we have seen a few of them requires more first-class citizen support with ReadingService, such as sharding, as they requires different manipulations depends on characteristics:

  • Filtering sharding: essentially replicate the upstream data (e.g. when the upstream data is still just metadata such as file list), and drop (N-1)/N of the data.
  • Round-robin sharding: to shard already-read data, replicating the upstream data pipe can be too expensive. In such case, we want to have different execution strategy that round robin the elements to different processes.

As such, we refactor to have _ShardingIterDataPipe as a parent class to indicate this special DataPipe in https://github.com/pytorch/pytorch/pull/94095 . As discussed in https://github.com/pytorch/pytorch/pull/94095#discussion_r1097710501, we found ShardingRoundRobinDispatcherIterDataPipe is a different sharding data pipe that has different execution strategy. We can also make it a subclass of _ShardingIterDataPipe and make related changes.

(More background to be added)

cc @ejguan , @kurman

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.