meta-pytorch / meta-pytorch/data
Definition of `IterDataPipe` in `pyi` file breaks inheritance path for static type checking
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
See comments in https://github.com/pytorch/data/pull/780
@pmeier
At list for the first Error, the proper typing should be:
def load(path: pathlib.Path) -> IterDataPipe[Tuple[str, BinaryIO]]:
if path.is_dir():
dp: IterDataPipe = FileLister(str(path), recursive=True)
else:
dp = IterableWrapper([str(path)])
return FileOpener(dp, mode="rb")
However, even with the proper typing shown above, the Error is changed to Incompatible types in assignment (expression has type "FileListerIterDataPipe", variable has type "IterDataPipe[Any]"). And, it doesn't explain what causes the second Error.
So, I spent a few hours figuring out what is the root cause of the mypy Error. In the generated datapipe.pyi file, a new IterDataPipe class is defined, which overrides the original IterDataPipe from the inheritance graph for all other DataPipe.
All Errors are eliminated when I remove new IterDataPipe definition from datapipe.pyi and import IterDataPipe directly from torch.utils.data.datapipe. And, the reason we define new IterDataPipe in pyi file is attaching all functional APIs to it. We need to do it in a different way by keeping the original IterDataPipe and extending the class with all functional APIs.
cc: @NivekT for python interface file
For this PR, I will revert it because our typing system needs to be fixed generally.
Originally posted by @ejguan in https://github.com/pytorch/data/issues/780#issuecomment-1252595095
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the generated datapipe.pyi file and the IterDataPipe definition imported from torch.utils.data.datapipe, then review the comments in pull request 780 and reproduce the reported mypy errors. The fix is complete when the original inheritance path is preserved, functional APIs remain available, and the shown typing errors are eliminated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100