apache / apache/airflow

🧟 Slow Zombie query causes scheduler heartbeat failures without errors

Open
#32,986 2 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:feature priority:low
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

2.6.3

### What happened

Due to an error with our postgres database (the stats on the tables were stale) this query:

https://github.com/apache/airflow/blob/1e20ef215ab8e688dc4331513fc5df34db443e84/airflow/jobs/scheduler_job_runner.py#L1686-L1698

took a very long time to return. During this time, heartbeats were not written, which caused health check failures (including k8s start / liveness check failures).

It took several days of debugging to track down the cause because airflow does not log any errors in this case. We resolved it by running `ANALYZE`.

### What you think should happen instead

Airflow should log warnings/errors if queries that are expected to return quickly take a long time to return.

### How to reproduce

1. Make your postgres database slow (either don't analyze statistics, or just change the query to something like `SELECT pg_sleep(2400);` for testing)
2. Try to run the airflow scheduler
3. Notice that heartbeats are not written frequently and no warnings or errors are logged

### Operating System

Debian GNU/Linux 11 (bullseye)

### Versions of Apache Airflow Providers

n/a

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow/jobs/scheduler_job_runner.py at the linked lines 1686-1698 and inspect the scheduler query involved in heartbeat updates. Reproduce the delay with a slow PostgreSQL query such as SELECT pg_sleep(2400), then determine the expected warning or error behavior. Done means a slow query produces useful scheduler logs instead of failing silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.