apache / apache/airflow

Task unexpectedly goes to up_for_retry with "Server indicated the task shouldn't be running anymore" and "Task killed!" errors

Open
#63,774 2 comments 0 reactions 0 assignees View on GitHub
area:core area:Executors-core kind:bug needs-triage priority:high
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

Other Airflow 3 version (please specify below)

### If "Other Airflow 3 version" selected, which one?

3.1.7

### What happened?

I created a DAG with a total of five tasks in offline environment.
When running, the first two tasks completed instantly within one or two seconds. However, when reaching the third task, it either shows up_for_retry without any log output, or runs for a short while and then shows up_for_retry.
The logs display: 'Server indicated the task shouldn't be running anymore', followed by Task not found, and the final line shows 'Task killed!'.

I also found this log in the scheduler logs.
Image

My configuration as below,
AIRFLOW_CORE__EXECUTOR: LocalExecutor
AIRFLOW_DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://airflow:airflow@mysql_airflow/airflow
AIRFLOW_CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW_CORE__LOAD_EXAMPLES: 'false'
AIRFLOW_API__SECRET_KEY: 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6'
AIRFLOW_API__SIMPLE_AUTH_MANAGER_USERS: 'admin:admin'
AIRFLOW_LOGGING__WORKER_LOG_SERVER_HOST: 'airflow-scheduler'
AIRFLOW_LOGGING__WORKER_LOG_SERVER_PORT: 8793
AIRFLOW_CORE__EXECUTION_API_SERVER_URL: 'http://airflow-api-server:8080/execution/'
AIRFLOW_API__AUTH__JWT_SECRET: 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6'
RAGFLOW_API_KEY: ragflow:ZNhD1iMmU4DCVjyEXZjE4MmZmNTZmMz
AIRFLOW_LOGGING__LOGGING_LEVEL: INFO
AIRFLOW_SCHEDULER__TASK_INSTANCE_HEARTBEAT_SEC: 30
AIRFLOW_SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT: 3000
AIRFLOW_SCHEDULER__TASK_INSTANCE_HEARTBEAT_DETECTION_INTERVAL: 30
AIRFLOW_SCHEDULER__SCHEDULER_HEARTBEAT_SEC: 20
AIRFLOW_SCHEDULER__ZOMBIE_DETECTION_INTERVAL: 300

### What you think should happen instead?

_No response_

### How to reproduce

we are in offline environment, met this prb. Other environment is normal.

### Operating System

rhel 9.7

### Versions of Apache Airflow Providers

Image

### Deployment

Other Docker-based deployment

### Deployment details

we used podman.

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.