Improve handling of Rendered Task Instance Fields
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Body
The current way how Rendered Task Instance Fields are handled (deleting old records and storing even big rendered task instance fields in the table) introduce a number of problems:
* Deadlocks on MySQL only
* https://github.com/apache/airflow/issues/32294
* https://github.com/apache/airflow/issues/29687
* https://github.com/apache/airflow/issues/18512
Those lead to workarounds such as #18616 and #32341
There are other reasons why big rendered fields should also be handled differently:
* https://github.com/apache/airflow/issues/28199
There was a discussion about deprecating it - that had never been concluded - but some good ideas and reasons why we can't **just** deprecate it have been explained there:
https://lists.apache.org/thread/jfzxq6xf8z5ss11hjl1yn41dd4mvk7cm
It would be great if someone picks up the discussion, lead it to conclusions and implement it.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by reviewing the linked deadlock and rendered-field issues, workarounds #18616 and #32341, and the Apache mailing-list discussion. Done means the discussion reaches a conclusion and the resulting implementation addresses deadlocks and the handling of large rendered task instance fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100