apache / apache/gravitino

[Improvement] Optimize the scheme of cloud storage

Open
#5,813 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

- Hadoop starts to support protocol `abfs` (abfss) for Azure cloud storage from 3.2.0, so if the version of Hadoop runtime is below 3.2.0, Gravitino can use GVFS client to access Azure cloud storage.
- We use `s3a://xxxx` as the scheme of AWS S3 storage, however, `s3a` is only the concept of Hadoop File System, it's not so proper to use it directly, so we'd better add support for `s3://xxxx` as well.

### How should we improve?

_No response_

Contributor guide

Open the contributing guide

Research direction

No files or tests are identified in the issue. Start by locating the Azure ABFS/GVFS and AWS S3A storage-scheme handling, then check how Hadoop runtime versions are detected. Done means Azure access works through GVFS below Hadoop 3.2.0 and AWS storage accepts both s3:// and s3a:// schemes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, azure, hadoop
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.