apache / apache/airflow

Airflow not publishing logs to Azure App Insights

Open
#72,093 2 comments 0 reactions 0 assignees View on GitHub
area:core area:logging kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.2.2

### What happened and how to reproduce it?

Upon migrating to airflow 3.2.2 from 2.9.3 my airflow application stopped publishing logs to Azure App Insights. Running Airlfow on AKS.
For 2.9.3 version I had written a custom handler and attached it to the airflow.task logger and that enabled the tunnel to push the airflow task logs to Azure app Insights in the CustomEvents table

Now using the similar approach for 3.2.2, the logs are not being pushed to AppInsights.
Library used for AppInsights connection: azure-monitor-opentelemetry==1.8.9, azure-monitor-opentelemetry-exporter==1.0.0b55.

I tried putting in logs for the handler initialisation- the initialsation logs were shown but the emit() logs never showed up in any of the pods

### What you think should happen instead?

_No response_

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by comparing the custom handler and airflow.task logger behavior between Airflow 2.9.3 and 3.2.2, then inspect pod logs to determine why initialization appears but emit() does not; done means task logs arrive in Azure App Insights CustomEvents.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes, python
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.