Airflow 3.2 Worker Failures on TaskInstance Finish (HTTP 409 invalid_state)
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
3.2.0. & 3.2.1
### What happened and how to reproduce it?
**Issue Description**
The observed behavior starts after upgrading to **Airflow 3.2.0** (and persists in **3.2.1**) and disappears when downgrading to **3.1.8**.
The failure manifests on Celery workers during the post-execution "finish" step (Task SDK / execution API), where the worker attempts to PATCH the TaskInstance state and receives:
- **HTTP 409**
- `reason=invalid_state`
- `message=TI was not in the running state so it cannot be updated`
- `previous_state=success`
This results in a worker-side exception (`ServerResponseError`) and causes the Celery workload to be treated as failed even when the underlying task logic has completed.
- Airflow version(s) affected: **3.2.0**, **3.2.1**
- Airflow version known-good: **3.1.8**
- Executor: **CeleryExecutor**
- Deployment: Kubernetes
- Python: 3.12
- Database: Postgres
Typical worker log sequence:
- Worker starts and loads secrets backend
- Worker begins Task SDK API calls (often with retries)
- API returns 409 invalid_state with `previous_state=success`
- Supervisor raises `ServerResponseError` and the Celery task fails
Example:
- `Secrets backends loaded for worker ...`
- `Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it.`
- `Starting call to ... this is the 2nd time calling it.`
- `API server error ... status_code=409 ... previous_state='success'`
- `airflow.sdk.api.client.ServerResponseError: Server returned error`
**Steps to reproduce**
The issue appears intermittently; higher concurrency tends to increase the frequency.
### Prerequisites
- Airflow **3.2.0+**
- **CeleryExecutor**
- Execution API enabled (Task SDK in use)
1. Deploy Airflow 3.2.0 or 3.2.1.
2. Trigger the DAG multiple times (or run multiple manual runs concurrently).
3. Observe worker logs for `ServerResponseError` with HTTP 409 `invalid_state` and `previous_state=success`.
### What you think should happen instead?
Airflow workers should be able to finalise a task idempotently; if the API server has already moved the TaskInstance to a terminal state (e.g. success), the worker’s PATCH /execution/task-instances//state should not return a fatal 409 or fail the task.
### Operating System
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.