apache / apache/datafusion-java

Spark DataSource backed by a DataFusion TableProvider over ADBC

オープン
#112 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Java
スター
32
フォーク
12
PR マージ指標
30日以内にマージされた PR はありません

説明

**Is your feature request related to a problem or challenge?**

Spark users want to read data from a DataFusion `TableProvider` as a native Spark `DataSourceV2`. Today there is no first-class path; options are either a bespoke per-operation JNI surface (more native surface to maintain) or copying data out of process.

**Describe the solution you'd like**

A Spark `DataSourceV2` connector that places the native boundary at a **standard ADBC driver**. Spark talks to the upstream arrow-adbc Java driver manager (`adbc-core` + `adbc-driver-jni`), which loads a native DataFusion ADBC cdylib and returns arrow-java `ArrowReader`s consumed zero-copy as `ArrowColumnVector`s on the cluster-provided Arrow. This reuses the upstream ADBC bindings rather than reproducing them.

Scope:
- `adbc-datafusion` format registered as a `DataSourceV2`; schema probed on the driver.
- Projection / filter / limit pushdown via Substrait, with a SQL fallback.
- Multi-partition reads (`executePartitioned` / `readPartition`) and a `target_partitions` option.
- Per-executor connection pool to amortize driver/database setup across task slots.
- An example DataFusion ADBC driver cdylib plus end-to-end (PySpark) coverage.

**Describe alternatives you've considered**

A plain-C scan ABI + hand-written JNI shim (discussed on #103 / #104). The ADBC approach reuses standard, separately-reviewed bindings and a stable driver contract instead.

**Additional context**

Implemented in #111.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start with the implementation referenced in #111, then compare it with this issue's stated scope: the adbc-datafusion DataSourceV2, pushdowns, partitioned reads, executor connection pooling, and PySpark coverage. Run the end-to-end coverage mentioned in the issue and verify that each listed capability is represented and working.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java, python, spark
領域
backend, data-engineering, distributed-systems
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
明確に書かれている
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。