meta-pytorch / meta-pytorch/data
Caching doesn't work with cycle
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
When cycle with later caching is used it works the very first time but afterwards it crashes because of the demux because it checks infinitely for todo files but all are cached.
from torchdata.datapipes.iter import IterableWrapper
dp = IterableWrapper(["test"])
dp = dp.cycle()
dp = dp.on_disk_cache(filepath_fn=lambda x: f"./{x}")
dp = dp.map(lambda x: (x, x))
dp = dp.end_caching(mode="t", same_filepath_fn=True)
next(iter(dp))
next(iter(dp))
This happens because of how demux works so the same applies for cycle and demux:
from torchdata.datapipes.iter import IterableWrapper
dp = IterableWrapper(["test"])
dp = dp.cycle()
dp0, dp1 = dp.demux(2, lambda x: 1)
next(iter(dp0))
Versions
torchdata.version=='0.7.0a0+deeacb4'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the two examples using IterableWrapper with cycle combined with on_disk_cache/end_caching and with demux. Read the cycle, demux, and caching DataPipe implementations to trace the repeated todo-file checks; done means repeated iteration no longer crashes or loops indefinitely when all items are cached.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100