apache / apache/airflow

KubernetesExecutor retries can lose failed-attempt hostname and remote logs, leading to http://:8793 No host supplied

Open
#65,366 1 comment 0 reactions 0 assignees View on GitHub
area:core area:logging kind:bug priority:high
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

3.1.8

### If "Other Airflow 3 version" selected, which one?

Also reproduced on 3.1.1.

### What happened?

We have multiple anonymized production incidents on Airflow 3.1.x with `KubernetesExecutor` where a task retries and the failed attempts are not retained correctly.

The visible UI symptom is:

```text
Could not read served logs: Invalid URL 'http://:8793/log/dag_id=.../run_id=.../task_id=.../attempt=3.log': No host supplied
```

But the more important observation is that the failed attempts already have broken metadata before log retrieval happens:

- the surviving `task_instance` row for the final successful retry is normal
- the corresponding `task_instance_history` rows for earlier failed attempts have `hostname=''`
- those same failed history rows also have `start_date = NULL`
- the remote log object for those failed attempts is missing from S3
- only the final successful attempt log object exists in remote storage

This was first seen on Airflow 3.1.1 and still reproduces on 3.1.8, so it does not look fixed by later 3.1 patch releases.

### What you think should happen instead?

When a task retries under `KubernetesExecutor`, each failed attempt should preserve normal historical metadata, including `hostname` and `start_date`, and its remote log object should remain accessible after later retries succeed.

The UI should not end up constructing `http://:8793/...` for historical attempts.

### How to reproduce

I do not yet have a minimal standalone reproducer, but the repeated field pattern is:

1. Run Airflow 3.1.x with `KubernetesExecutor`
2. Enable remote S3 logging
3. Run a DAG where a task fails, retries, and later succeeds
4. Inspect historical task attempts for that task/run

Observed result in affected runs:

- final `task_instance` row is normal and points to the last successful attempt
- earlier `task_instance_history` rows for failed attempts have blank hostname and null start_date
- remote log objects for those failed attempts do not exist
- only the final successful attempt log object exists

### Operating System

Linux / Kubernetes

### Versions of Apache Airflow Providers

Not yet isolated to a provider-specific issue. The symptom has appeared on normal task retries for dbt-related DAG tasks with remote S3 logging enabled.

### Deployment

Other Kubernetes deployment

### Deployment details

Environment characteristics shared by affected runs:

- Executor: `KubernetesExecutor`
- Remote logging: enabled to S3
- Airflow versions: reproduced on 3.1.1 and 3.1.8
- Multiple scheduler replicas were present in at least one affected environment

This does **not** look like the Celery/proxy issue in #64263. In these incidents:

- there is no Celery worker layer involved
- successful sibling tasks in the same DAG run do persist logs normally
- final successful retries for the same task also persist logs normally
- only the earlier failed attempts are missing both hostname metadata and remote log objects

### Anything else?

A concrete anonymized example from Airflow 3.1.8:

For one task that eventually succeeded on `try_number=8`, the database state looked like this:

```text
final task_instance row:
try_number=8, state='success', start_date=2026-04-16 08:13:48+00:00, hostname='10.x.x.x'

task_instance_history rows:
try_number=3, state='failed', start_date=NULL, hostname=''
try_number=6, state='failed', start_date=NULL, hostname=''
```

Remote logging for that same task/run contained only:

```text
attempt=8.log
```

and direct checks for `attempt=3.log` and `attempt=6.log` returned `404`.

We saw the same pattern on other tasks in the same DAG run, not just a single task.

The UI error seems to be a downstream consequence of the blank hostname. `FileTaskHandler` later tries to build the served-log URL from `ti.hostname`, which explains the final `http://:8793/...` message, but by that point the task-attempt metadata and remote log object are already missing.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.