meta-pytorch / meta-pytorch/data
Prevent (or at least flag) DataPipe attributes and methods that use existing functional datapipe names
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🚀 Feature
We should have some checks/tests that flag when a DataPipe has an attribute/method that shares the same name as existing functional datapipes. Those names are the ones defined inside the decorator @functional_datapipe('NAME'), such as map, batch, zip, and etc.
For example, Batcher (or BatcherIterDataPipe) has the functional datapipe name batch. However, currently there is nothing to prevent other IterDataPipes to use batch as the name of an attribute or method.
The change in the following PR is a good example.
Motivation
If this feature is not implemented, then a DataPipe can have multiple attributes/methods with the same name, potentially causing confusion and bugs.
Alternatives
Ideally, we should be able to flag this issue during development (within IDEs).
If we cannot automatically prevent this during development, we can have a check in register_datapipe_as_function or a CI check that ensures all attributes and methods are compliant.
mypy also should be able to flag this issue if the .pyi file has a complete set of method interfaces for all built-in DataPipes (including those in TorchData). This becomes trickier for user-defined DataPipes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with register_datapipe_as_function and the @functional_datapipe definitions that establish names such as map, batch, and zip. Review the existing DataPipe interfaces in the .pyi file and consider the registration, CI, and mypy alternatives described here. Done means collisions between functional names and DataPipe attributes or methods are reliably flagged, with checks or tests covering the relevant cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data-engineering, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100