apache / apache/gravitino

[FEATURE] Add JuiceFS support in Fileset

Open
#4,359 6 comments 0 reactions 0 assignees View on GitHub
epic
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 15h
Merged PRs (30d)
315

Description

### Describe the feature

Fileset is a new concept introduced in 0.5.0 to manage non-tabular data; the current implementation uses HCFS to manage physical data. Now, HCFS doesn't support JuiceFS.

In this issue, we should discuss: how to support JuiceFS in Fileset and how to achieve it.

### Motivation

JuiceFS is a high-performance, cloud-native distributed file system that is developing rapidly. Support of this could help Gravitino to be used in more scenarios in the future

### Describe the solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Fileset implementation and its current HCFS-based physical-data management, then determine how JuiceFS would fit into that design. The issue provides no file paths or tests; done would require an agreed implementation approach and demonstrated JuiceFS support in Fileset.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.