apache / apache/datafusion-python

Wire FFI_TableProvider / TableProvider(ABC) through to the Rust TableProvider trait

Open
#1,668 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
604
Forks
174
Avg merge
1d 7h
Merged PRs (30d)
4

Description

Split out from #1577 (item 2).

Unlike `CatalogProvider` / `SchemaProvider`, wiring a plain-Python `TableProvider` subclass through to the Rust `TableProvider` trait needs a new `RustWrappedPyTableProvider`, analogous to `RustWrappedPySchemaProvider` in `catalog.rs`. It has to cover `scan()` returning an `ExecutionPlan`, `supports_filters_pushdown`, and statistics.

This is execution-engine work, not a typing-only change, so it doesn't belong in a typing PR. Opening as its own issue so the design (partial pushdown support, how statistics get surfaced to the Python side) can be discussed before someone picks it up.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading catalog.rs, especially RustWrappedPySchemaProvider, and review item 2 from #1577. Trace the TableProvider(ABC) and Rust TableProvider entry points, then clarify how scan(), supports_filters_pushdown, and statistics should cross the boundary. Done means the wrapper and its partial-pushdown and statistics behavior are implemented and agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.