apache / apache/fluss

[filesystem] Add JuiceFS filesystem support

Open
#3,866 0 comments 0 reactions 1 assignee Claimed by @weizuo93 View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

[JuiceFS](https://github.com/juicedata/juicefs) is a high-performance distributed file system designed for cloud-native scenarios, which can be connected to various object storage services at the underlying layer. Some enterprises adopt JuiceFS as a unified access layer for various object storage services. It ships with a Hadoop-compatible Java SDK, making it well-suited to serve as remote storage for snapshots of primary-key tables and tiered log segments of log tables within Fluss.

### Solution

Add a new filesystem plugin module `fluss-fs-juicefs` under `fluss-filesystems/`, registered under the `jfs://` URI scheme, bridging Fluss to the JuiceFS Hadoop SDK.

* **Module**: `fluss-filesystems/fluss-fs-juicefs`
* **Scheme**: `jfs://`
* **Underlying SDK**: `io.juicefs:juicefs-hadoop`
* **Distribution**: Not bundled in the default binary distribution; installed manually into `${FLUSS_HOME}/plugins/juicefs/`, consistent with the COS / OBS / Azure plugins.

## Design / Implementation Overview

The plugin is intentionally thin and reuses `HadoopFileSystem` from `fluss-fs-hadoop`, mirroring the structure of the existing OSS / COS plugins.

1. **`JuiceFsPlugin implements FileSystemPlugin`**
* `getScheme()` returns `"jfs"`.
* `create(URI, Configuration)` builds a Hadoop `Configuration` from Fluss config, applies JuiceFS defaults, and instantiates the JuiceFS Hadoop `FileSystem` via reflection (`FileSystem.newInstance`) so that Fluss has no hard compile-time dependency on `io.juicefs.*`.
* Forwards any Fluss configuration entry whose key starts with `fs.jfs.` or `juicefs.` to the Hadoop `Configuration`.
* Injects sensible defaults when the user does not set them:
* `fs.jfs.impl = io.juicefs.JuiceFileSystem`
* `fs.jfs.impl.disable.cache = false`

2. **`JuiceFsFileSystem extends HadoopFileSystem`**
* Wraps the JuiceFS Hadoop `FileSystem`.
* Overrides `obtainSecurityToken()` to return an **empty placeholder token**, because JuiceFS handles authentication locally (see below).

3. **SPI registration** via `META-INF/services/org.apache.fluss.fs.FileSystemPlugin`.

4. **Shading / packaging** consistent with the other filesystem plugins (depends on `fluss-fs-hadoop-shaded`, `fluss-fs-hadoop`, and the `juicefs-hadoop` SDK).

5. **Tests**: A `JuiceFsPluginTest` covering scheme registration, prefix-based config forwarding (`fs.jfs.*`, `juicefs.*`, and unrelated keys), default-value injection (and non-override of user values), and null-config safety.

### Configuration Example

Minimum required configuration in `server.yaml`:
```
# The dir that used to be as the remote storage of Fluss
remote.data.dir: jfs:///path/to/remote/storage

# JuiceFS metadata engine address of the pre-created volume, e.g. redis://:/
juicefs.meta:
```

Any other `fs.jfs.*` / `juicefs.*` keys documented in the [JuiceFS Hadoop Java SDK docs](https://juicefs.com/docs/community/hadoop_java_sdk) can be added to `server.yaml` and will be transparently forwarded, e.g.:
```
juicefs.cache-dir: /data*/jfscache
juicefs.cache-size: 1024
juicefs.access-log: /tmp/juicefs.access.log
```

### Authentication Model

Unlike the OSS / S3 / COS plugins, Fluss does **not** perform any STS or delegation-token exchange for JuiceFS:
* The JuiceFS client on each node authenticates directly against its metadata engine and backing object storage, using credentials embedded when the JuiceFS volume was formatted (or supplied via `juicefs.access-key` / `juicefs.secret-key` overrides where applicable).
* **Deployment implication**: every Fluss process that accesses remote storage (CoordinatorServer, TabletServer, and every client) must have (a) the plugin installed, (b) the same `juicefs.*` configuration, and (c) network reachability to the JuiceFS metadata engine and object storage.

### References

* JuiceFS Hadoop Java SDK: https://juicefs.com/docs/community/hadoop_java_sdk
* JuiceFS GitHub: https://github.com/juicedata/juicefs

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.