apache / apache/datafusion-java

Publish fat JAR with platform-specific native libraries to Maven Central

Ouverte
#33 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Java
Étoiles
32
Forks
12
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

## Background

`datafusion-java` provides a JVM binding to DataFusion via JNI. To distribute it through Maven Central, we need a packaging strategy that delivers the compiled Rust native library (`.so` / `.dylib` / `.dll`) alongside the Java classes so that consumers get a working artifact with a single dependency declaration — no separate native install step.

## Goal

Publish a single artifact to Maven Central that works out of the box on:
- Linux x86_64
- Linux aarch64
- macOS x86_64
- macOS aarch64

Windows (x86_64) support is desirable but out of scope for the initial release. The design should leave room to add it later without restructuring.

## Proposed approach: single fat JAR

Bundle all platform-specific native libraries in one published JAR, organized by OS/arch under a known resource path:

```
org/apache/datafusion/linux/amd64/libdatafusion_jni.so
org/apache/datafusion/linux/aarch64/libdatafusion_jni.so
org/apache/datafusion/darwin/x86_64/libdatafusion_jni.dylib
org/apache/datafusion/darwin/aarch64/libdatafusion_jni.dylib
```

At runtime, a loader class detects the current OS/arch, extracts the matching library from the JAR to a temp file, and calls `System.load()` on the absolute path. A `System.loadLibrary()` attempt should come first so users can override with a system-installed build.

This mirrors the approach used by Apache DataFusion Comet (referenced only as prior art for fat-JAR packaging — `datafusion-java` is not otherwise related to Comet or Spark). The alternative — publishing one JAR per platform with Maven classifiers — is also viable but pushes platform selection onto consumers and complicates dependency declarations.

## Work items

- [ ] Add a native loader class that detects OS/arch, extracts from the resource path, and loads via `System.load()`, with a `System.loadLibrary()` fallback. Include temp-file locking to handle concurrent JVMs.
- [ ] Set up cross-compilation for the four target triples (Linux x86_64, Linux aarch64, macOS x86_64, macOS aarch64). Options: a CI matrix that produces per-arch artifacts, or Docker + OSXCross for cross-platform builds from a single host.
- [ ] Wire the build so compiled libraries land at the correct `target/classes/...` path before `mvn package` runs.
- [ ] Add a GitHub Actions release workflow: matrix builds per platform produce native libs as artifacts; a final job assembles them into the resource tree and runs `mvn deploy`.
- [ ] Configure Maven Central / Sonatype publishing: staging repo, GPG signing, POM metadata.
- [ ] Document the release process.

## Future work

- [ ] Windows x86_64 support. The loader OS enum should already account for `.dll` and the `win32` path segment so this becomes a build-matrix change. Windows complicates temp-file cleanup (can't delete a loaded DLL) — extract to a versioned path and let the OS handle cleanup.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.