apache / apache/airflow

Improve handling of Rendered Task Instance Fields

Open
#32,342 4 comments 1 reaction 0 assignees View on GitHub
affected_version:main_branch area:MetaDB kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Body

The current way how Rendered Task Instance Fields are handled (deleting old records and storing even big rendered task instance fields in the table) introduce a number of problems:

* Deadlocks on MySQL only
* https://github.com/apache/airflow/issues/32294
* https://github.com/apache/airflow/issues/29687
* https://github.com/apache/airflow/issues/18512

Those lead to workarounds such as #18616 and #32341

There are other reasons why big rendered fields should also be handled differently:

* https://github.com/apache/airflow/issues/28199

There was a discussion about deprecating it - that had never been concluded - but some good ideas and reasons why we can't **just** deprecate it have been explained there:

https://lists.apache.org/thread/jfzxq6xf8z5ss11hjl1yn41dd4mvk7cm

It would be great if someone picks up the discussion, lead it to conclusions and implement it.

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked deadlock and rendered-field issues, workarounds #18616 and #32341, and the Apache mailing-list discussion. Done means the discussion reaches a conclusion and the resulting implementation addresses deadlocks and the handling of large rendered task instance fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.