apache / apache/airflow

Should we handle `asset_state` cleanup when `clear_on_success` is set?

Open
#67,228 0 comments 0 reactions 1 assignee Claimed by @amoghrajesh View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

Ref: https://github.com/apache/airflow/pull/66859#discussion_r3270160482

What i think:
I was under the impression that "task states" only need to be cleared for clear_on_success because those are bound to tasks and asset states are intentionally not cleared on task success, server-side or worker-side cos asset state is designed to persist across task runs (e.g. watermarks, file counts that accumulate), so clearing it on success would not be beneficial by default. If there's a case for clearing asset state on success in the future, it should be a separate opt-in config (e.g. [state_store] clear_asset_state_on_success) as per me.

cc @kaxil

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.