apache / apache/airflow

Cannot deactivate File-based logs for at processor

Open
#70,968 3 comments 1 reaction 0 assignees View on GitHub
area:logging kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.3.0

### What happened and how to reproduce it?

## Goal

Completely disable file-based logs from dag processor.

## Context

I Run airflow. I Notice that dag_processor creates many logs under `/opt/airflow/logs/dag_processor`. In fact, this directory eventually grows multiple GBs large and eats all available space.

I can create some log rotator / archiver, but in fact I do not need the `dag_processor` logs at all so **I would very much prefer to deactivate them completely**.

## What I tried

1. Setting `[logging] --> dag_processor_log_target = stdout` and restarted airflow. Intuitively I would expect that this would work, but unfortunately the directory still being created and filled with multiple MBs per day.

A sample from the files I see:

```
{"timestamp":"2026-08-03T00:23:28.159066Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:23:28.163495Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
{"timestamp":"2026-08-03T00:23:58.281367Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:23:58.286175Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
{"timestamp":"2026-08-03T00:24:28.471662Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:24:28.475547Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
{"timestamp":"2026-08-03T00:24:59.339487Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/purchases/purchases_batch_prices_sync.py","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","logger":"airflow.dag_processing.dagbag.BundleDagBag","filename":"dagbag.py","lineno":460}
{"timestamp":"2026-08-03T00:24:59.343819Z","level":"warning","event":"The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html","dag_file":"purchases/purchases_batch_prices_sync.py","bundle_name":"dags-folder","category":"UserWarning","filename":"","lineno":96,"logger":"py.warnings"}
```

### What you think should happen instead?

Setting `dag_processor_log_target = stdout`should meain no logs written to the file system

### Operating System

docker container

### Deployment

Docker-Compose

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

Only mounting some volumes (/opt/airflow/logs/ being one of them) and pip installing "apache-airflow-providers-oracle"

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the [logging] dag_processor_log_target setting through the dag processor logging setup, reproducing the behavior with the Docker-Compose deployment described here. Done means selecting stdout prevents dag processor log files from being created or populated under /opt/airflow/logs/dag_processor while the process remains usable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.