meta-pytorch / meta-pytorch/data

Prevent (or at least flag) DataPipe attributes and methods that use existing functional datapipe names

Open
#101 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🚀 Feature

We should have some checks/tests that flag when a DataPipe has an attribute/method that shares the same name as existing functional datapipes. Those names are the ones defined inside the decorator @functional_datapipe('NAME'), such as map, batch, zip, and etc.

For example, Batcher (or BatcherIterDataPipe) has the functional datapipe name batch. However, currently there is nothing to prevent other IterDataPipes to use batch as the name of an attribute or method.

The change in the following PR is a good example.

Motivation

If this feature is not implemented, then a DataPipe can have multiple attributes/methods with the same name, potentially causing confusion and bugs.

Alternatives

Ideally, we should be able to flag this issue during development (within IDEs).

If we cannot automatically prevent this during development, we can have a check in register_datapipe_as_function or a CI check that ensures all attributes and methods are compliant.

mypy also should be able to flag this issue if the .pyi file has a complete set of method interfaces for all built-in DataPipes (including those in TorchData). This becomes trickier for user-defined DataPipes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with register_datapipe_as_function and the @functional_datapipe definitions that establish names such as map, batch, and zip. Review the existing DataPipe interfaces in the .pyi file and consider the registration, CI, and mypy alternatives described here. Done means collisions between functional names and DataPipe attributes or methods are reliably flagged, with checks or tests covering the relevant cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data-engineering, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.