[Feature] Support for azure datalake (abfs)
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
I'm new to Paimon, I found Paimon does not support Azure blob and Azure datalake yet. We use Azure Blob on the iceberg in our company, and I'm familiar with Azure Blob and Azure datalake, so I want to implement it. I'd like to implement [abfs(abfss)filesystem] (https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver) just like oss and s3 fileIO
What do you think about this? @JingsongLi
### Solution
Implement the azure abfs filesystem just like s3 and oss.
https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing S3 and OSS FileIO implementations and the Hadoop Azure DataLake documentation linked in the issue. Confirm the integration supports both ABFS and ABFSS filesystem access in the same way as the existing filesystem integrations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, hadoop, java
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100