task_instance_heartbeat_timeout is ignored when >2
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Apache Airflow version
Other Airflow 2/3 version (please specify below)
### If "Other Airflow 2/3 version" selected, which one?
3.0.6
### What happened?
i have a test sleep-10-min DAG.
my settings were:
task_instance_heartbeat_sec = 0
**task_instance_heartbeat_timeout = 300**
task_instance_heartbeat_timeout_detection_interval = 10.0
i was running this DAG with values 300, 1, 2, 4, 5, 6, 10. between runs airflow containers were respawned.
except values 1, 2 DAG was not terminated.
for the cases 1 and 2 duration were around 9 - 19 seconds.
### What you think should happen instead?
tasks shoud be terminated after task_instance_heartbeat_timeout.
### How to reproduce
DAG:
`from airflow.sdk import dag
from airflow.providers.standard.operators.bash import BashOperator
from datetime import datetime
@dag(start_date=datetime(2021, 1, 1), schedule="@once", catchup=False)
def sleep_dag():
t1 = BashOperator(
task_id="sleep_10_minutes",
bash_command="sleep 600",
)
sleep_dag()
`
be sure you have no further settings override on docker-compose.yaml
set settings in airflow.cfg to section scheduler:
my settings were:
task_instance_heartbeat_sec = 0
**task_instance_heartbeat_timeout = 300**
task_instance_heartbeat_timeout_detection_interval = 10.0
### Operating System
Ubuntu 24.04.3 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Reproduce the sleep_10_minutes DAG with the scheduler settings shown in airflow.cfg and the Docker-Compose deployment, testing timeout values above 2. Start by tracing the scheduler's heartbeat-timeout handling; done means tasks terminate after task_instance_heartbeat_timeout for values such as 300, not only 1 or 2.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100