Introduce a logging remote task handler for webhdfs
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
We've been using WebHDFS as remote logging solution. The solution we use is adjusted from the [S3TaskHandler](https://github.com/apache/airflow/blob/4e545c6e54712eedb6ca9cbb8333393ae3f6cba2/airflow/providers/amazon/aws/log/s3_task_handler.py). Since the `apache-webhdfs`-provider is already available and the project already depends on `hdfs`, I think this should be fairly forward contribution.
Before I go ahead and create an actual pull-request, is there anything I should keep in mind with regards to this contribution?
### Use case/motivation
I'm interested into contributing our internal `WebHDFSTaskHandler`, as I feel the community could benefit.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reading airflow/providers/amazon/aws/log/s3_task_handler.py, the referenced model for this contribution, and review how the existing apache-webhdfs provider and hdfs dependency are used. Done should be a WebHDFS remote logging task handler integrated into Airflow with coverage for its expected logging behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100