apache / apache/airflow

Duplicate rows on rendered_task_instance_fields

Open
#53,905 23 comments 10 reactions 0 assignees View on GitHub
affected_version:3.0 area:core area:MetaDB kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

3.0.3

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

Here is the error on the first try. But on the second try, the task was successful with no error. While I get the error and it is still running/trying the DAG is not shown in the UI anymore. The problem happens randomly on any task of the DAG. The task is Kubernetes POD operator calling a Python module under docker. It's on AKS and the database of Airflow is Postgres.

Task failed with exception: source="task"
AirflowRuntimeError: API_SERVER_ERROR: {'status_code': 409, 'message': 'Server returned error', 'detail': {'detail': {'reason': 'Unique constraint violation', 'statement': 'INSERT INTO rendered_task_instance_fields (dag_id, task_id, run_id, map_index, rendered_fields, k8s_pod_yaml) VALUES (%(dag_id)s, %(task_id)s, %(run_id)s, %(map_index)s, %(rendered_fields)s, %(k8s_pod_yaml)s)', 'orig_error': 'duplicate key value violates unique constraint "rendered_task_instance_fields_pkey"\nDETAIL: Key (dag_id, task_id, run_id, map_index)=(sdds_dag_view_snapshot, task_group3.snapshot_START_sec_bond, manual__2025-07-29T21:54:12.221693+00:00, -1) already exists.\n', 'message': 'Serious error when handling your request. Check logs for more details - you will find it in api server when you look for ID R8X88i8e'}}}

### What you think should happen instead?

It should not get that error. Also the DAG has to keep presenting in the UI since it is parsed correctly.

### How to reproduce

Not sure if anyone can reproduce it since it is random but let me know if more info needed. Thanks

### Operating System

Azure - AKS running container / docker.

### Versions of Apache Airflow Providers

3.0.3 (latest version)

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the API server handling of inserts into rendered_task_instance_fields and the Kubernetes pod task's rendering path, using the reported PostgreSQL unique-constraint error as the entry point. Investigate why the same dag_id, task_id, run_id, and map_index can be inserted twice; done means the duplicate failure is prevented and the DAG remains visible in the UI during task execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, docker, kubernetes, postgres, python
Domain
backend, data-engineering, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.