meta-pytorch / meta-pytorch/data
Add cache invalidation to OnDiskCacheHolder
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🚀 The feature
Add the ability to automatically invalidate a cached sub-graph when the remote files change after being cached locally.
Motivation, pitch
Say I have multiple files stored in a remote object storage. These files are fed into a datapipe using FSSpecFileLister, and cached locally using .on_disk_cache. I want to invalidate the cache and re-compute the datapipe when one or more remote files are changed, probably based on their hash.
Alternatives
No response
Additional context
This feature request originated from this conversation on the Pytorch forum.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading OnDiskCacheHolder and the FSSpecFileLister usage described in the issue to understand how cached sub-graphs are created. Determine how remote file changes should be detected, then verify that changed files invalidate the local cache and cause the datapipe to recompute.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100