meta-pytorch / meta-pytorch/data
What does it mean for a DataPipe to be 'replicable'?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
📚 The doc issue
In the ReadingService docs the different sharding options and that one applies to replicable and one to non-replicable datapipes, but it's not really explained what that means.
Indirectly related, I'm also confused by the names ShardingRoundRobinDispatcher and ShardingFilter. The docs for ShardingFilter say
each instance of the DataPipe (on different workers) will have every n-th element of the original DataPipe, where n equals to the number of instances.
Is that not essentially the definition of round robin distribution? How is that different than what the the DataPipes downstream of a ShardingRoundRobinDispatcher on different workers receive?
Suggest a potential alternative/fix
Clarify more the difference between ShardingRoundRobinDispatcher and ShardingFilter and explain what 'replicable' means in that context.
Possibly consider renaming ShardingRoundRobinDispatcher and ShardingFilter, if the answers to my questions above are 'yes' to something more meaningful.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked ReadingService documentation and trace the definitions or usages of ShardingRoundRobinDispatcher and ShardingFilter. Determine how the two sharding options differ and what replicable means in this context. Done means the documentation clearly explains these terms and their worker-level behavior, including whether the names are misleading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100