apache / apache/datafusion-java

Spark DataSource backed by a DataFusion TableProvider over ADBC

Open
#112 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
32
Forks
12
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem or challenge?**

Spark users want to read data from a DataFusion `TableProvider` as a native Spark `DataSourceV2`. Today there is no first-class path; options are either a bespoke per-operation JNI surface (more native surface to maintain) or copying data out of process.

**Describe the solution you'd like**

A Spark `DataSourceV2` connector that places the native boundary at a **standard ADBC driver**. Spark talks to the upstream arrow-adbc Java driver manager (`adbc-core` + `adbc-driver-jni`), which loads a native DataFusion ADBC cdylib and returns arrow-java `ArrowReader`s consumed zero-copy as `ArrowColumnVector`s on the cluster-provided Arrow. This reuses the upstream ADBC bindings rather than reproducing them.

Scope:
- `adbc-datafusion` format registered as a `DataSourceV2`; schema probed on the driver.
- Projection / filter / limit pushdown via Substrait, with a SQL fallback.
- Multi-partition reads (`executePartitioned` / `readPartition`) and a `target_partitions` option.
- Per-executor connection pool to amortize driver/database setup across task slots.
- An example DataFusion ADBC driver cdylib plus end-to-end (PySpark) coverage.

**Describe alternatives you've considered**

A plain-C scan ABI + hand-written JNI shim (discussed on #103 / #104). The ADBC approach reuses standard, separately-reviewed bindings and a stable driver contract instead.

**Additional context**

Implemented in #111.

Contributor guide

Open the contributing guide

Research direction

Start with the implementation referenced in #111, then compare it with this issue's stated scope: the adbc-datafusion DataSourceV2, pushdowns, partitioned reads, executor connection pooling, and PySpark coverage. Run the end-to-end coverage mentioned in the issue and verify that each listed capability is represented and working.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, spark
Domain
backend, data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.