Add support for Azure Data Lake Storage as a Apache Beam FileSystem
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
This is for providing direct integration with Azure Data Lake Store as an Apache Beam File system.
There is already support for Azure Data Lake for using it as HDFS: https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html
Imported from Jira [BEAM-3038](https://issues.apache.org/jira/browse/BEAM-3038). Original Jira may contain additional context.
Reported by: datageekromit.
Contributor guide
Research direction
Start by reviewing the original Jira ticket BEAM-3038 and the Hadoop Azure Data Lake documentation linked in the issue to understand the existing HDFS-based support. Done means Apache Beam has direct Azure Data Lake Storage integration as a FileSystem, but the issue does not name implementation files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100