apache / apache/datafusion-python
Wire FFI_TableProvider / TableProvider(ABC) through to the Rust TableProvider trait
- Dominant language
- Python
- Stars
- 604
- Forks
- 174
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 4
Description
Split out from #1577 (item 2).
Unlike `CatalogProvider` / `SchemaProvider`, wiring a plain-Python `TableProvider` subclass through to the Rust `TableProvider` trait needs a new `RustWrappedPyTableProvider`, analogous to `RustWrappedPySchemaProvider` in `catalog.rs`. It has to cover `scan()` returning an `ExecutionPlan`, `supports_filters_pushdown`, and statistics.
This is execution-engine work, not a typing-only change, so it doesn't belong in a typing PR. Opening as its own issue so the design (partial pushdown support, how statistics get surfaced to the Python side) can be discussed before someone picks it up.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading catalog.rs, especially RustWrappedPySchemaProvider, and review item 2 from #1577. Trace the TableProvider(ABC) and Rust TableProvider entry points, then clarify how scan(), supports_filters_pushdown, and statistics should cross the boundary. Done means the wrapper and its partial-pushdown and statistics behavior are implemented and agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100