apache / apache/datafusion-java

Publish fat JAR with platform-specific native libraries to Maven Central

未關閉
#33 2 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Java
星號
32
分支
12
PR 合併指標
30 天內沒有已合併 PR

描述

## Background

`datafusion-java` provides a JVM binding to DataFusion via JNI. To distribute it through Maven Central, we need a packaging strategy that delivers the compiled Rust native library (`.so` / `.dylib` / `.dll`) alongside the Java classes so that consumers get a working artifact with a single dependency declaration — no separate native install step.

## Goal

Publish a single artifact to Maven Central that works out of the box on:
- Linux x86_64
- Linux aarch64
- macOS x86_64
- macOS aarch64

Windows (x86_64) support is desirable but out of scope for the initial release. The design should leave room to add it later without restructuring.

## Proposed approach: single fat JAR

Bundle all platform-specific native libraries in one published JAR, organized by OS/arch under a known resource path:

```
org/apache/datafusion/linux/amd64/libdatafusion_jni.so
org/apache/datafusion/linux/aarch64/libdatafusion_jni.so
org/apache/datafusion/darwin/x86_64/libdatafusion_jni.dylib
org/apache/datafusion/darwin/aarch64/libdatafusion_jni.dylib
```

At runtime, a loader class detects the current OS/arch, extracts the matching library from the JAR to a temp file, and calls `System.load()` on the absolute path. A `System.loadLibrary()` attempt should come first so users can override with a system-installed build.

This mirrors the approach used by Apache DataFusion Comet (referenced only as prior art for fat-JAR packaging — `datafusion-java` is not otherwise related to Comet or Spark). The alternative — publishing one JAR per platform with Maven classifiers — is also viable but pushes platform selection onto consumers and complicates dependency declarations.

## Work items

- [ ] Add a native loader class that detects OS/arch, extracts from the resource path, and loads via `System.load()`, with a `System.loadLibrary()` fallback. Include temp-file locking to handle concurrent JVMs.
- [ ] Set up cross-compilation for the four target triples (Linux x86_64, Linux aarch64, macOS x86_64, macOS aarch64). Options: a CI matrix that produces per-arch artifacts, or Docker + OSXCross for cross-platform builds from a single host.
- [ ] Wire the build so compiled libraries land at the correct `target/classes/...` path before `mvn package` runs.
- [ ] Add a GitHub Actions release workflow: matrix builds per platform produce native libs as artifacts; a final job assembles them into the resource tree and runs `mvn deploy`.
- [ ] Configure Maven Central / Sonatype publishing: staging repo, GPG signing, POM metadata.
- [ ] Document the release process.

## Future work

- [ ] Windows x86_64 support. The loader OS enum should already account for `.dll` and the `win32` path segment so this becomes a build-matrix change. Windows complicates temp-file cleanup (can't delete a loaded DLL) — extract to a versioned path and let the OS handle cleanup.

貢獻指南

開啟貢獻指南

研究方向

先檢查現有的 JNI 與 Maven 建置設定,然後執行 mvn package 以了解目前的 artifact 配置。追蹤如何為列出的四個目標產生 native library 並組裝到 target/classes 下;完成的定義是單一 Maven Central artifact 載入相符的 library,且 release workflow 以有文件記錄的步驟部署它。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
github-actions, java, rust
領域
build-system, ci-cd, release
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
冷清
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。