[Improvement] Optimize the scheme of cloud storage
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
- Hadoop starts to support protocol `abfs` (abfss) for Azure cloud storage from 3.2.0, so if the version of Hadoop runtime is below 3.2.0, Gravitino can use GVFS client to access Azure cloud storage.
- We use `s3a://xxxx` as the scheme of AWS S3 storage, however, `s3a` is only the concept of Hadoop File System, it's not so proper to use it directly, so we'd better add support for `s3://xxxx` as well.
### How should we improve?
_No response_
Contributor guide
Research direction
No files or tests are identified in the issue. Start by locating the Azure ABFS/GVFS and AWS S3A storage-scheme handling, then check how Hadoop runtime versions are detected. Done means Azure access works through GVFS below Hadoop 3.2.0 and AWS storage accepts both s3:// and s3a:// schemes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure, hadoop
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100