meta-pytorch / meta-pytorch/data
Returned "path" of `HTTPReader` and `GDriveReader` diverges
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
HTTPReader returns the URL for the "path"
while GDriveReader returns the file name
Since OnlineReader determines at runtime whether to call the HTTP or GDrive download
the "path" of the yielded tuples is impossible to predict:
from torchdata.datapipes.iter import IterableWrapper, OnlineReader
dp = IterableWrapper(
[
"https://raw.githubusercontent.com/pytorch/data/main/LICENSE",
"https://drive.google.com/uc?export=download&id=1GO-BHUYRuvzr1Gtp2_fqXRsr9TIeYbhV",
]
)
dp = OnlineReader(dp)
for path, _ in dp:
print(path)
https://raw.githubusercontent.com/pytorch/data/main/LICENSE
torchvision.txt
We should align the two. My vote is out to align based on the file name. Still, returning the URL could also be useful if redirect logic as discussed in https://github.com/pytorch/vision/pull/6060#pullrequestreview-980646299 is added to the HTTPReader.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in torchdata/datapipes/iter/load/online.py at HTTPReader, GDriveReader, and OnlineReader. Compare the path values yielded by the two readers and how OnlineReader selects between them, then resolve which path semantics should be consistent. Reproduce the example in the issue and verify that both reader types yield the agreed form.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100