meta-pytorch / meta-pytorch/data

What does it mean for a DataPipe to be 'replicable'?

Open
#1,131 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

📚 The doc issue

In the ReadingService docs the different sharding options and that one applies to replicable and one to non-replicable datapipes, but it's not really explained what that means.

Indirectly related, I'm also confused by the names ShardingRoundRobinDispatcher and ShardingFilter. The docs for ShardingFilter say

each instance of the DataPipe (on different workers) will have every n-th element of the original DataPipe, where n equals to the number of instances.

Is that not essentially the definition of round robin distribution? How is that different than what the the DataPipes downstream of a ShardingRoundRobinDispatcher on different workers receive?

Suggest a potential alternative/fix

Clarify more the difference between ShardingRoundRobinDispatcher and ShardingFilter and explain what 'replicable' means in that context.

Possibly consider renaming ShardingRoundRobinDispatcher and ShardingFilter, if the answers to my questions above are 'yes' to something more meaningful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked ReadingService documentation and trace the definitions or usages of ShardingRoundRobinDispatcher and ShardingFilter. Determine how the two sharding options differ and what replicable means in this context. Done means the documentation clearly explains these terms and their worker-level behavior, including whether the names are misleading.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.