apache / apache/airflow

Airflow 3.2 Worker Failures on TaskInstance Finish (HTTP 409 invalid_state)

Open
#65,708 5 comments 8 reactions 0 assignees View on GitHub
area:core kind:bug needs-triage priority:high
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Task SDK

### Apache Airflow version

3.2.0. & 3.2.1

### What happened and how to reproduce it?

**Issue Description**

The observed behavior starts after upgrading to **Airflow 3.2.0** (and persists in **3.2.1**) and disappears when downgrading to **3.1.8**.

The failure manifests on Celery workers during the post-execution "finish" step (Task SDK / execution API), where the worker attempts to PATCH the TaskInstance state and receives:

- **HTTP 409**
- `reason=invalid_state`
- `message=TI was not in the running state so it cannot be updated`
- `previous_state=success`

This results in a worker-side exception (`ServerResponseError`) and causes the Celery workload to be treated as failed even when the underlying task logic has completed.

- Airflow version(s) affected: **3.2.0**, **3.2.1**
- Airflow version known-good: **3.1.8**
- Executor: **CeleryExecutor**
- Deployment: Kubernetes
- Python: 3.12
- Database: Postgres

Typical worker log sequence:

- Worker starts and loads secrets backend
- Worker begins Task SDK API calls (often with retries)
- API returns 409 invalid_state with `previous_state=success`
- Supervisor raises `ServerResponseError` and the Celery task fails

Example:

- `Secrets backends loaded for worker ...`
- `Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it.`
- `Starting call to ... this is the 2nd time calling it.`
- `API server error ... status_code=409 ... previous_state='success'`
- `airflow.sdk.api.client.ServerResponseError: Server returned error`

**Steps to reproduce**

The issue appears intermittently; higher concurrency tends to increase the frequency.

### Prerequisites
- Airflow **3.2.0+**
- **CeleryExecutor**
- Execution API enabled (Task SDK in use)

1. Deploy Airflow 3.2.0 or 3.2.1.
2. Trigger the DAG multiple times (or run multiple manual runs concurrently).
3. Observe worker logs for `ServerResponseError` with HTTP 409 `invalid_state` and `previous_state=success`.

### What you think should happen instead?

Airflow workers should be able to finalise a task idempotently; if the API server has already moved the TaskInstance to a terminal state (e.g. success), the worker’s PATCH /execution/task-instances//state should not return a fatal 409 or fail the task.

### Operating System

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.