Should we handle `asset_state` cleanup when `clear_on_success` is set?
Open
area:core
kind:feature
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
Ref: https://github.com/apache/airflow/pull/66859#discussion_r3270160482
What i think:
I was under the impression that "task states" only need to be cleared for clear_on_success because those are bound to tasks and asset states are intentionally not cleared on task success, server-side or worker-side cos asset state is designed to persist across task runs (e.g. watermarks, file counts that accumulate), so clearing it on success would not be beneficial by default. If there's a case for clearing asset state on success in the future, it should be a separate opt-in config (e.g. [state_store] clear_asset_state_on_success) as per me.
cc @kaxil
Contributor guide
Assessment
This issue has not been assessed yet.