apache / apache/airflow

Present task-errors caused by scheduler in UI

Open
#28,116 5 comments 0 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Some times tasks are marked as failed by the scheduler or other internal airflow components, with the reason being known by the scheduler, but not surfaced to users. Instead the task is just marked as failed, without any indication why in the Logs-tab nor under Task instance details. Only way to figure out is for admins to go through the scheduler-logs.

Example: `{scheduler_job.py:1526} ERROR - Detected zombie job: .....`, marks the task as failed.
Here the UI looks like this, with the logs-tab being only showing an empty attempt of fetching log files from worker:
![image](https://user-images.githubusercontent.com/89977373/205640822-6c3f05e3-3c15-4dbd-b46b-5f415bbf1962.png)

If there is a need to hide potential scheduler-secrets from users, there should at least be some generic error message saying that the task was not run to completion because of internal scheduler error. So that users know if they should debug task-logic themselves, or involve admins in debugging the infrastructure.

Looking at `taskinstance.handle_failure`-function it seem like most the code for handling such errors is already very centralized. Is there a way to attach the error there onto the taskinstance itself, for use in the UI? Or is it possible to merge the relevant log streams to show in the Logs-tab?

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading taskinstance.handle_failure and scheduler_job.py around line 1526, then trace how failed task logs and Task instance details are populated in the UI. Define whether scheduler-caused errors should appear in the Logs tab, task details, or both, while keeping sensitive scheduler information hidden. Done means users can distinguish internal scheduler failures from task-logic failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.