apache / apache/gravitino

Cache Hadoop Filesystem instance on Gravitino server to improve the performance

Open
#6,561 5 comments 0 reactions 1 assignee Claimed by @sunxiaojian View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

Currently , all Gravitino File system providers use the following code

(Take HDFSFileSystemProvider for example)

Image

`FileSystem.instance` will always create a new Filesystem everytime even though they are the same mostly. In fact Hadoop FileSystem did have cache mechanism, If we use `FileSystem.get`, cache mechanism in FileSystem will works. Due to the fact the Gravitino virtual FileSystem (GVFS) client also shares FileSystemProviders and supports credentials for each unique path, we should be cautious when planning to enable cache in the file system. in all
- In Gravitno server side, we can enable cache in FileSystem level
- In GVFS, we need to disable it FileSystem level and cache file system instacen in GVFS level

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.