meta-pytorch / meta-pytorch/data
[RFC] Iterating over MapDataPipes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🚀 The feature
For context please read https://github.com/pytorch/data/issues/795 first.
Iterating over MapDatapipes is currently inconsistent, we should find a way to resolve this.
Motivation, pitch
source_dp = IterableWrapper([(i, i) for i in range(10)])
map_dp = source_dp.to_map_datapipe()
print(list(map_dp))
> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
works as expected but
source_dp = IterableWrapper([(i+1, i) for i in range(10)])
map_dp = source_dp.to_map_datapipe()
print(list(map_dp))
> []
does not.
Alternatives
Do we want to overwrite iter for MapDatapipes? We already have to_iter_datapipe. If yes:
- Should we return the keys or the values?
- Should we add a warning that this is not the intended way?
- ...
Additional context
This post explains how python iterates over classes that overwrite getitem but not iter:
https://stackoverflow.com/questions/68244987/how-do-dunder-methods-getitem-and-len-provide-iteration
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue #795 and the MapDataPipes examples using IterableWrapper and to_map_datapipe(). Compare iteration for keys beginning at 0 with keys beginning at 1, then resolve and document the intended iteration semantics, including whether iteration returns keys or values and whether a warning is needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100