apache / apache/airflow

Remote logging provider support for separate reader and (multiple) writer(s)

Open
#48,874 1 comment 1 reaction 0 assignees View on GitHub
area:logging kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

The newly introduced `RemoteLogIO`, moves the abstraction for remote logging from the `logging.Handler` to purely `read` and `upload`, let's use this ticket to formalize the interface for remote logging solutions in Airflow 3.x.

### Use case/motivation

Most handlers provided by providers implement `logging.Handler` have adopted this pattern. Our use-case is to ship logs to Kafka and read from ElasticSearch. We need to separate the reading and writing in order to keep control over the ingestion speed in a multi-tenant ES cluster.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading the RemoteLogIO abstraction and its relationship to logging.Handler. Compare the stated Kafka ingestion and ElasticSearch reading use case with the proposed read and upload separation. Done means Airflow 3.x has a formal remote-logging interface that supports separate readers and multiple writers, with the expected scope agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, kafka, python
Domain
backend, data-engineering, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.