[Core feature] Allow users to control storage for caching
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 120
Description
### Motivation: Why do you think this is important?
I am using Flyte to process large datasets (couple GB) and I enabled caching for my tasks. However, I noticed my storage quickly filled up because old cache entries were not deleted.
### Goal: What should the final outcome look like, ideally?
I propose the following features:
- Users should be able to configure a limit (at least a soft limit) to the amount of storage consumed by the cache. Cache entries should then be evicted using a least-recently-used policy (or maybe something clever - many small cache entries might be more important than one large?).
- The Flyte API (and CLI) should expose a way to clear the cache (and the cache for individual tasks, projects, ...)
### Describe alternatives you've considered
AFAIK, there is currently no way to clear the cache.
### Propose: Link/Inline OR Additional context
_No response_
### Are you sure this issue hasn't been raised already?
- [x] Yes
### Have you read the Code of Conduct?
- [x] Yes
Contributor guide
Research direction
Start by tracing Flyte's existing cache implementation and the API and CLI entry points mentioned in the issue. Determine how cache entries are stored, whether size metadata and eviction hooks already exist, and how task- or project-level clearing could be exposed. Done means the storage limit, eviction behavior, and cache-clearing scope are defined and covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100