Remote logging provider support for separate reader and (multiple) writer(s)
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
The newly introduced `RemoteLogIO`, moves the abstraction for remote logging from the `logging.Handler` to purely `read` and `upload`, let's use this ticket to formalize the interface for remote logging solutions in Airflow 3.x.
### Use case/motivation
Most handlers provided by providers implement `logging.Handler` have adopted this pattern. Our use-case is to ship logs to Kafka and read from ElasticSearch. We need to separate the reading and writing in order to keep control over the ingestion speed in a multi-tenant ES cluster.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reading the RemoteLogIO abstraction and its relationship to logging.Handler. Compare the stated Kafka ingestion and ElasticSearch reading use case with the proposed read and upload separation. Done means Airflow 3.x has a formal remote-logging interface that supports separate readers and multiple writers, with the expected scope agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, kafka, python
- Domain
- backend, data-engineering, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100