[Core feature] Support cache overwrite flag at task level
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 120
Description
### Motivation: Why do you think this is important?
Currently, when tasks need to bypass cache and force re-execution, developers need to modify the `cache_version` parameter. However, this is an internal implementation detail that shouldn't be exposed to end users. Users should have a more intuitive way to control cache behavior at runtime.
### Goal: What should the final outcome look like, ideally?
## Proposal
Add support for a `cache_overwrite` or `force_recompute` flag at the task level that can be set during execution. This would allow:
1. Task definition remains unchanged
2. Users can control cache behavior at runtime
3. Clear separation between developer concerns (`cache_version`) and user control
### Describe alternatives you've considered
1. Workflow-Level Cache Control
- Add cache control parameters at workflow level only
- Pros: Simpler API surface, clearer execution tracking
- Cons: Too coarse-grained, can't control individual tasks
2. Using Cache Version
- Current approach: Modify cache_version
- Pros: Already implemented
- Cons: Mixes developer and user concerns, requires code changes
### Propose: Link/Inline OR Additional context
Example API could look like:
```python
@task(cache=True, overwrite_cache=False) # Default behavior
def my_task():
pass
# At Runtime
wf.my_task(overwrite_cache=True) # Force recompute
```
Implementation (rough idea):
- Add one more field called `overwrite_cache` for Flyte Task Decorator
- Add helper func: `IsTaskCacheOverwrite` for `cache.go`
- In `executor.go`, check if `IsTaskCacheOverwrite`, if yes, return `CatalogCacheStatus_CACHE_SKIPPED`
### Are you sure this issue hasn't been raised already?
- [X] Yes
### Have you read the Code of Conduct?
- [X] Yes
Contributor guide
Assessment
This issue has not been assessed yet.