Task unexpectedly goes to up_for_retry with "Server indicated the task shouldn't be running anymore" and "Task killed!" errors
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
Other Airflow 3 version (please specify below)
### If "Other Airflow 3 version" selected, which one?
3.1.7
### What happened?
I created a DAG with a total of five tasks in offline environment.
When running, the first two tasks completed instantly within one or two seconds. However, when reaching the third task, it either shows up_for_retry without any log output, or runs for a short while and then shows up_for_retry.
The logs display: 'Server indicated the task shouldn't be running anymore', followed by Task not found, and the final line shows 'Task killed!'.
I also found this log in the scheduler logs.
My configuration as below,
AIRFLOW_CORE__EXECUTOR: LocalExecutor
AIRFLOW_DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://airflow:airflow@mysql_airflow/airflow
AIRFLOW_CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW_CORE__LOAD_EXAMPLES: 'false'
AIRFLOW_API__SECRET_KEY: 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6'
AIRFLOW_API__SIMPLE_AUTH_MANAGER_USERS: 'admin:admin'
AIRFLOW_LOGGING__WORKER_LOG_SERVER_HOST: 'airflow-scheduler'
AIRFLOW_LOGGING__WORKER_LOG_SERVER_PORT: 8793
AIRFLOW_CORE__EXECUTION_API_SERVER_URL: 'http://airflow-api-server:8080/execution/'
AIRFLOW_API__AUTH__JWT_SECRET: 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6'
RAGFLOW_API_KEY: ragflow:ZNhD1iMmU4DCVjyEXZjE4MmZmNTZmMz
AIRFLOW_LOGGING__LOGGING_LEVEL: INFO
AIRFLOW_SCHEDULER__TASK_INSTANCE_HEARTBEAT_SEC: 30
AIRFLOW_SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT: 3000
AIRFLOW_SCHEDULER__TASK_INSTANCE_HEARTBEAT_DETECTION_INTERVAL: 30
AIRFLOW_SCHEDULER__SCHEDULER_HEARTBEAT_SEC: 20
AIRFLOW_SCHEDULER__ZOMBIE_DETECTION_INTERVAL: 300
### What you think should happen instead?
_No response_
### How to reproduce
we are in offline environment, met this prb. Other environment is normal.
### Operating System
rhel 9.7
### Versions of Apache Airflow Providers
### Deployment
Other Docker-based deployment
### Deployment details
we used podman.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.