meta-pytorch / meta-pytorch/data

Definition of `IterDataPipe` in `pyi` file breaks inheritance path for static type checking

Open
#782 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Better Engineering
Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

See comments in https://github.com/pytorch/data/pull/780

@pmeier
At list for the first Error, the proper typing should be:

def load(path: pathlib.Path) -> IterDataPipe[Tuple[str, BinaryIO]]:
    if path.is_dir():
        dp: IterDataPipe = FileLister(str(path), recursive=True)
    else:
        dp = IterableWrapper([str(path)])
    return FileOpener(dp, mode="rb")

However, even with the proper typing shown above, the Error is changed to Incompatible types in assignment (expression has type "FileListerIterDataPipe", variable has type "IterDataPipe[Any]"). And, it doesn't explain what causes the second Error.
So, I spent a few hours figuring out what is the root cause of the mypy Error. In the generated datapipe.pyi file, a new IterDataPipe class is defined, which overrides the original IterDataPipe from the inheritance graph for all other DataPipe.

All Errors are eliminated when I remove new IterDataPipe definition from datapipe.pyi and import IterDataPipe directly from torch.utils.data.datapipe. And, the reason we define new IterDataPipe in pyi file is attaching all functional APIs to it. We need to do it in a different way by keeping the original IterDataPipe and extending the class with all functional APIs.

cc: @NivekT for python interface file

For this PR, I will revert it because our typing system needs to be fixed generally.

Originally posted by @ejguan in https://github.com/pytorch/data/issues/780#issuecomment-1252595095

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the generated datapipe.pyi file and the IterDataPipe definition imported from torch.utils.data.datapipe, then review the comments in pull request 780 and reproduce the reported mypy errors. The fix is complete when the original inheritance path is preserved, functional APIs remain available, and the shown typing errors are eliminated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.