apache / apache/airflow

task_instance_heartbeat_timeout is ignored when >2

Open
#58,307 3 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

Other Airflow 2/3 version (please specify below)

### If "Other Airflow 2/3 version" selected, which one?

3.0.6

### What happened?

i have a test sleep-10-min DAG.
my settings were:
task_instance_heartbeat_sec = 0
**task_instance_heartbeat_timeout = 300**
task_instance_heartbeat_timeout_detection_interval = 10.0

i was running this DAG with values 300, 1, 2, 4, 5, 6, 10. between runs airflow containers were respawned.
except values 1, 2 DAG was not terminated.
for the cases 1 and 2 duration were around 9 - 19 seconds.

### What you think should happen instead?

tasks shoud be terminated after task_instance_heartbeat_timeout.

### How to reproduce

DAG:
`from airflow.sdk import dag
from airflow.providers.standard.operators.bash import BashOperator
from datetime import datetime

@dag(start_date=datetime(2021, 1, 1), schedule="@once", catchup=False)
def sleep_dag():
t1 = BashOperator(
task_id="sleep_10_minutes",
bash_command="sleep 600",
)

sleep_dag()
`
be sure you have no further settings override on docker-compose.yaml
set settings in airflow.cfg to section scheduler:
my settings were:
task_instance_heartbeat_sec = 0
**task_instance_heartbeat_timeout = 300**
task_instance_heartbeat_timeout_detection_interval = 10.0

### Operating System

Ubuntu 24.04.3 LTS

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Reproduce the sleep_10_minutes DAG with the scheduler settings shown in airflow.cfg and the Docker-Compose deployment, testing timeout values above 2. Start by tracing the scheduler's heartbeat-timeout handling; done means tasks terminate after task_instance_heartbeat_timeout for values such as 300, not only 1 or 2.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.