[filesystem] Add JuiceFS filesystem support
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
[JuiceFS](https://github.com/juicedata/juicefs) is a high-performance distributed file system designed for cloud-native scenarios, which can be connected to various object storage services at the underlying layer. Some enterprises adopt JuiceFS as a unified access layer for various object storage services. It ships with a Hadoop-compatible Java SDK, making it well-suited to serve as remote storage for snapshots of primary-key tables and tiered log segments of log tables within Fluss.
### Solution
Add a new filesystem plugin module `fluss-fs-juicefs` under `fluss-filesystems/`, registered under the `jfs://` URI scheme, bridging Fluss to the JuiceFS Hadoop SDK.
* **Module**: `fluss-filesystems/fluss-fs-juicefs`
* **Scheme**: `jfs://`
* **Underlying SDK**: `io.juicefs:juicefs-hadoop`
* **Distribution**: Not bundled in the default binary distribution; installed manually into `${FLUSS_HOME}/plugins/juicefs/`, consistent with the COS / OBS / Azure plugins.
## Design / Implementation Overview
The plugin is intentionally thin and reuses `HadoopFileSystem` from `fluss-fs-hadoop`, mirroring the structure of the existing OSS / COS plugins.
1. **`JuiceFsPlugin implements FileSystemPlugin`**
* `getScheme()` returns `"jfs"`.
* `create(URI, Configuration)` builds a Hadoop `Configuration` from Fluss config, applies JuiceFS defaults, and instantiates the JuiceFS Hadoop `FileSystem` via reflection (`FileSystem.newInstance`) so that Fluss has no hard compile-time dependency on `io.juicefs.*`.
* Forwards any Fluss configuration entry whose key starts with `fs.jfs.` or `juicefs.` to the Hadoop `Configuration`.
* Injects sensible defaults when the user does not set them:
* `fs.jfs.impl = io.juicefs.JuiceFileSystem`
* `fs.jfs.impl.disable.cache = false`
2. **`JuiceFsFileSystem extends HadoopFileSystem`**
* Wraps the JuiceFS Hadoop `FileSystem`.
* Overrides `obtainSecurityToken()` to return an **empty placeholder token**, because JuiceFS handles authentication locally (see below).
3. **SPI registration** via `META-INF/services/org.apache.fluss.fs.FileSystemPlugin`.
4. **Shading / packaging** consistent with the other filesystem plugins (depends on `fluss-fs-hadoop-shaded`, `fluss-fs-hadoop`, and the `juicefs-hadoop` SDK).
5. **Tests**: A `JuiceFsPluginTest` covering scheme registration, prefix-based config forwarding (`fs.jfs.*`, `juicefs.*`, and unrelated keys), default-value injection (and non-override of user values), and null-config safety.
### Configuration Example
Minimum required configuration in `server.yaml`:
```
# The dir that used to be as the remote storage of Fluss
remote.data.dir: jfs:///path/to/remote/storage
# JuiceFS metadata engine address of the pre-created volume, e.g. redis://:/
juicefs.meta:
```
Any other `fs.jfs.*` / `juicefs.*` keys documented in the [JuiceFS Hadoop Java SDK docs](https://juicefs.com/docs/community/hadoop_java_sdk) can be added to `server.yaml` and will be transparently forwarded, e.g.:
```
juicefs.cache-dir: /data*/jfscache
juicefs.cache-size: 1024
juicefs.access-log: /tmp/juicefs.access.log
```
### Authentication Model
Unlike the OSS / S3 / COS plugins, Fluss does **not** perform any STS or delegation-token exchange for JuiceFS:
* The JuiceFS client on each node authenticates directly against its metadata engine and backing object storage, using credentials embedded when the JuiceFS volume was formatted (or supplied via `juicefs.access-key` / `juicefs.secret-key` overrides where applicable).
* **Deployment implication**: every Fluss process that accesses remote storage (CoordinatorServer, TabletServer, and every client) must have (a) the plugin installed, (b) the same `juicefs.*` configuration, and (c) network reachability to the JuiceFS metadata engine and object storage.
### References
* JuiceFS Hadoop Java SDK: https://juicefs.com/docs/community/hadoop_java_sdk
* JuiceFS GitHub: https://github.com/juicedata/juicefs
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.