apache / apache/airflow

Bug with the advanced logging configuration documentation with remote logging and custom logging config class for airflow version 3.2.2

Open Beginner friendly
#71,768 2 comments 0 reactions 0 assignees View on GitHub
area:logging kind:bug kind:documentation needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Documentation

### Apache Airflow version

3.2.2

### What happened and how to reproduce it?

I have got an Airflow self hosted deployment on Azure Kubernetes cluster with remote logging enabled to azure blob storage. I was migrating my existing deployment from airflow version 2.8.4 to 3.2.2.

My setup involves `remote_logging` enabled with logs copied to azure blob storage. I use a custom `logging_config_class`. This was all working on airflow version 2.8.4.

I followed what was mentioned in the documentation of version [3.2.2 advanced logging configuration](https://airflow.apache.org/docs/apache-airflow/3.2.2/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html).

After migrating to v3.2.2 I was getting this warning in the scheduler logs
> warning ] remote_log_handler_unavailable [airflow.logging.remote] loc=log.py:236 note='Remote log handler could not be loaded; logs will be available locally only.'

And the logs were not copied to my azure blob storage container.

Upon going through the source code for the remote logging to work, I found that we need to set the following global variables in the python module.

```
REMOTE_TASK_LOG: RemoteLogIO | None = MyRemoteLogIO()
DEFAULT_REMOTE_CONN_ID: str | None = "my_remote_conn"
```

This wasn't captured in the airflow 3.2.2 documentation and after fixing this, the scheduler warning related to remote logging disappeared and the remote logs were written to azure blob storage.

### What you think should happen instead?

The documentation for the advanced logging that uses custom logging_config_class along with remote logging enabled scenario is not captured correctly.

The documentation for airflow version 3.3.1 is updated with this information, but the documentation for airflow version 3.2.2 should be updated with this information.

### Operating System

NAME="Debian GNU/Linux", VERSION_ID="12", VERSION_CODENAME=bookworm

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

```
apache-airflow-providers-amazon 9.29.0
apache-airflow-providers-apache-livy 4.5.6
apache-airflow-providers-cncf-kubernetes 10.17.1
apache-airflow-providers-common-compat 1.15.0
apache-airflow-providers-common-io 1.7.2
apache-airflow-providers-common-sql 2.0.0
apache-airflow-providers-fab 3.6.4
apache-airflow-providers-http 6.0.2
apache-airflow-providers-microsoft-azure 13.3.0
apache-airflow-providers-postgres 6.7.0
apache-airflow-providers-smtp 3.0.1
apache-airflow-providers-ssh 5.0.2
apache-airflow-providers-standard 1.13.1
```

### Official Helm Chart version

1.22.0 (latest released)

### Kubernetes Version

1.35.3

### Helm Chart configuration

Airflow logging related YAML configuration
```YAML
logging:
remote_logging: "True"
remote_log_conn_id: "azblob_logging"
remote_base_log_folder: "wasb://wasb-airflow-logs-aks"
base_log_folder: "/opt/airflow/logs"
logging_level: "INFO"
fab_logging_level: "WARN"
colored_console_log: "True"
colored_formatter_class: "airflow.utils.log.colored_log.CustomTTYColoredFormatter"
simple_log_format: "%%(asctime)s %%(levelname)s - %%(message)s"
dag_processor_child_process_log_directory: "/opt/airflow/logs/dag_processor"
task_log_reader: "task"
encrypt_s3_logs: "False"
delete_local_logs: "False"
logging_config_class: "config.log_config.LOGGING_CONFIG"

azure_remote_logging:
remote_wasb_log_container: "airflow-logs"

```

### Docker Image customizations

Not Applicable

### Anything else?

This was my original logging configuration

```python
from copy import deepcopy

from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG

LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)

# Prevent task log records from propagating to the root console handler.
# The default propagate=True means every record is handled twice (FileTaskHandler
# + root console), which the Airflow UI merges into duplicate log lines.
LOGGING_CONFIG["loggers"]["airflow.task"]["propagate"] = False
```

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Compare the Airflow 3.2.2 advanced logging configuration page linked in the issue with the updated 3.3.1 documentation, focusing on custom logging_config_class with remote logging. Document the required REMOTE_TASK_LOG and DEFAULT_REMOTE_CONN_ID configuration, and confirm that the 3.2.2 page explains how Azure remote logs become available.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, helm, kubernetes, python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.