apache / apache/airflow

Introduce a logging remote task handler for webhdfs

Open
#28,608 1 comment 1 reaction 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

We've been using WebHDFS as remote logging solution. The solution we use is adjusted from the [S3TaskHandler](https://github.com/apache/airflow/blob/4e545c6e54712eedb6ca9cbb8333393ae3f6cba2/airflow/providers/amazon/aws/log/s3_task_handler.py). Since the `apache-webhdfs`-provider is already available and the project already depends on `hdfs`, I think this should be fairly forward contribution.

Before I go ahead and create an actual pull-request, is there anything I should keep in mind with regards to this contribution?

### Use case/motivation

I'm interested into contributing our internal `WebHDFSTaskHandler`, as I feel the community could benefit.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading airflow/providers/amazon/aws/log/s3_task_handler.py, the referenced model for this contribution, and review how the existing apache-webhdfs provider and hdfs dependency are used. Done should be a WebHDFS remote logging task handler integrated into Airflow with coverage for its expected logging behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.