meta-pytorch / meta-pytorch/data
Caching will not unblock pipes if there is a filter between `on_disk_cache` and `end_caching`
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
See https://github.com/pytorch/text/issues/1735
Filtering of untar results leads to empty datapipe and block readers. Looks like we need to lint and probably work around this situation in cache too.
Versions
Master
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the on_disk_cache and end_caching behavior, then consult the linked pytorch/text issue 1735 and reproduce the filtered untar case. Trace how an empty datapipe blocks readers and identify the lint or cache workaround needed. Done means filtered results no longer leave readers blocked when caching is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100