apache / apache/airflow

Don't retry failures because of execution timeouts

Open
#9,232 1 comment 14 reactions 0 assignees View on GitHub
area:core area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

When reaching execution_timeout, a task fails but continues to retry. Either it should not be retried in this situation, or there should be a possibility to define another timeout for the "total" task execution, taking into account all retries.

**Use case / motivation**

In our case current behavior makes execution_timeout feature useless: we have retries in place to prevent random issues like network connectivity. At the same time, we want to make sure that the tasks don't run for too long and execution_time + retries would just make them running even longer.

See also https://stackoverflow.com/questions/53830604/airflow-execution-timeout-resetting-every-retry: one more request for the same feature.

Contributor guide

Open the contributing guide

Research direction

The issue names execution_timeout, task retries, and total task execution as the relevant entry points, but no files or tests. Start by tracing how execution_timeout interacts with retry handling, then clarify which proposed behavior is required. Done means the selected timeout policy is implemented and verified for tasks that would otherwise retry after timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.