Cannot deactivate File-based logs for at processor
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.0
### What happened and how to reproduce it?
## Goal
Completely disable file-based logs from dag processor.
## Context
I Run airflow. I Notice that dag_processor creates many logs under `/opt/airflow/logs/dag_processor`. In fact, this directory eventually grows multiple GBs large and eats all available space.
I can create some log rotator / archiver, but in fact I do not need the `dag_processor` logs at all so **I would very much prefer to deactivate them completely**.
## What I tried
1. Setting `[logging] --> dag_processor_log_target = stdout` and restarted airflow. Intuitively I would expect that this would work, but unfortunately the directory still being created and filled with multiple MBs per day.
A sample from the files I see:
```
{"timestamp":"2026-08-03T00:23:28.159066Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:23:28.163495Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
{"timestamp":"2026-08-03T00:23:58.281367Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:23:58.286175Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
{"timestamp":"2026-08-03T00:24:28.471662Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:24:28.475547Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
{"timestamp":"2026-08-03T00:24:59.339487Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:24:59.343819Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
```
### What you think should happen instead?
Setting `dag_processor_log_target = stdout`should meain no logs written to the file system
### Operating System
docker container
### Deployment
Docker-Compose
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
Only mounting some volumes (/opt/airflow/logs/ being one of them) and pip installing "apache-airflow-providers-oracle"
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by tracing the [logging] dag_processor_log_target setting through the dag processor logging setup, reproducing the behavior with the Docker-Compose deployment described here. Done means selecting stdout prevents dag processor log files from being created or populated under /opt/airflow/logs/dag_processor while the process remains usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100