apache / apache/paimon

[Feature] Support for azure datalake (abfs)

Open
#2,700 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

I'm new to Paimon, I found Paimon does not support Azure blob and Azure datalake yet. We use Azure Blob on the iceberg in our company, and I'm familiar with Azure Blob and Azure datalake, so I want to implement it. I'd like to implement [abfs(abfss)filesystem] (https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver) just like oss and s3 fileIO

What do you think about this? @JingsongLi

### Solution

Implement the azure abfs filesystem just like s3 and oss.
https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing S3 and OSS FileIO implementations and the Hadoop Azure DataLake documentation linked in the issue. Confirm the integration supports both ABFS and ABFSS filesystem access in the same way as the existing filesystem integrations.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, hadoop, java
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.