meta-pytorch / meta-pytorch/data
Add documentation about custom Shuffle and Sharding DataPipe
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
📚 The doc issue
TorchData has a few special graph functions to handle Shuffle and Sharding DataPipe. But, we never document what is expected for those graph functions, which leads users to extend custom shuffle and sharding by diving into our code base.
We should add clear document about the expected methods attached to Shuffle or Sharding DataPipe.
This problem has been discussed in the #1081 as well, but I want to track documentation issue separately
Suggest a potential alternative/fix
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing TorchData's existing documentation and the graph functions for Shuffle and Sharding DataPipe, then read the related discussion in issue #1081. Document the methods expected for custom Shuffle and Sharding DataPipe implementations, with enough detail that users no longer need to inspect the codebase.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100