Intermittent DAG Task Failures with Missing Logs ("No host supplied") using CeleryExecutor
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
3.1.0
### What happened?
### Deployment Details
- Executor: CeleryExecutor
- Installation method: Installed via pip (not Docker or Kubernetes)
### Issue Description
Tasks occasionally fail immediately at the first task without producing any logs. The Airflow UI shows the following error:
`Could not read served logs: Invalid URL 'http://:8793/log/dag_id=.../task_id=.../attempt=1.log': No host supplied`
After restarting Airflow services (webserver and scheduler), the problem temporarily resolves, but it reoccurs after days.
### Impact
This issue is intermittent but recurring, making it difficult to rely on Airflow for production workloads. Even when tasks appear to succeed after a service restart, the problem resurfaces after some days, causing sudden DAG failures with no logs
### Observations / Notes
- Restarting services temporarily fixes the issue.
- The problem seems related to hostname resolution or worker URL caching in the webserver.
- I monitored memory usage and noticed that the issue appears when system memory is nearly full — it seems that hostnames or related metadata are lost or not resolved correctly when memory pressure increases.
- The issue occurs intermittently and is hard to reproduce consistently, but becomes more likely over time as memory usage grows.
### What you think should happen instead?
The Airflow webserver should consistently resolve and serve valid log URLs for all tasks, regardless of uptime or restart state.
Tasks should not fail immediately due to missing or malformed log endpoints, and restarts should not be required to restore normal behavior.
### Operating System
Ubuntu 24.04.2 LTS
### Versions of Apache Airflow Providers
3.0.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by investigating log serving between the webserver and CeleryExecutor workers, focusing on how worker hostnames and URLs are resolved after long uptime and under memory pressure. Reproduce the intermittent “No host supplied” error if possible; done means task logs remain accessible through valid URLs without requiring service restarts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100