apache / apache/airflow

Scheduler can lose executor callback events on database commit failure

Open
#69,975 0 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

`main`

### What happened and how to reproduce it?

The Scheduler drains each executor's event buffer before the database transaction that records the events has committed. If that commit raises a transient database error, an executor callback completion event has already been removed from the buffer even though its `Callback` row remains in `QUEUED` or `RUNNING`.

Unlike task instances, executor callbacks have no recovery path that reselects these states. The callback can therefore remain active indefinitely.

To reproduce on `main`:

1. Create an `ExecutorCallback` in `QUEUED` state.
2. Add a matching successful `CallbackKey` event to an executor's event buffer.
3. Run the Scheduler's executor-event processing while making the event session's `commit()` raise an `OperationalError`.
4. Observe that the callback row is still `QUEUED`, while the original event buffer is empty.

This was reproduced against the production event-processing path with an actual transaction rollback.

### What you think should happen instead?

Executor callback completion events should remain available until their state update commits. Transient DB errors should be retried without replaying task-event side effects. If the configured DB retries are exhausted, the callback events should return to their owning executor buffer and be retried by the next Scheduler loop without terminating the Scheduler.

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at the Scheduler's executor-event processing path and reproduce the failure with an ExecutorCallback, a successful CallbackKey event, and a commit() that raises OperationalError. Done means events remain available until the state update commits, exhausted retries restore them to the owning executor buffer, and the Scheduler continues running without replaying task-event side effects.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.