Duplicate rows on rendered_task_instance_fields
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Apache Airflow version
3.0.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Here is the error on the first try. But on the second try, the task was successful with no error. While I get the error and it is still running/trying the DAG is not shown in the UI anymore. The problem happens randomly on any task of the DAG. The task is Kubernetes POD operator calling a Python module under docker. It's on AKS and the database of Airflow is Postgres.
Task failed with exception: source="task"
AirflowRuntimeError: API_SERVER_ERROR: {'status_code': 409, 'message': 'Server returned error', 'detail': {'detail': {'reason': 'Unique constraint violation', 'statement': 'INSERT INTO rendered_task_instance_fields (dag_id, task_id, run_id, map_index, rendered_fields, k8s_pod_yaml) VALUES (%(dag_id)s, %(task_id)s, %(run_id)s, %(map_index)s, %(rendered_fields)s, %(k8s_pod_yaml)s)', 'orig_error': 'duplicate key value violates unique constraint "rendered_task_instance_fields_pkey"\nDETAIL: Key (dag_id, task_id, run_id, map_index)=(sdds_dag_view_snapshot, task_group3.snapshot_START_sec_bond, manual__2025-07-29T21:54:12.221693+00:00, -1) already exists.\n', 'message': 'Serious error when handling your request. Check logs for more details - you will find it in api server when you look for ID R8X88i8e'}}}
### What you think should happen instead?
It should not get that error. Also the DAG has to keep presenting in the UI since it is parsed correctly.
### How to reproduce
Not sure if anyone can reproduce it since it is random but let me know if more info needed. Thanks
### Operating System
Azure - AKS running container / docker.
### Versions of Apache Airflow Providers
3.0.3 (latest version)
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by tracing the API server handling of inserts into rendered_task_instance_fields and the Kubernetes pod task's rendering path, using the reported PostgreSQL unique-constraint error as the entry point. Investigate why the same dag_id, task_id, run_id, and map_index can be inserted twice; done means the duplicate failure is prevented and the DAG remains visible in the UI during task execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, docker, kubernetes, postgres, python
- Domain
- backend, data-engineering, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100