apache / apache/datafusion

FFI: `FFI_ExecutionPlan` silently drops producer overrides of optimizer-relevant defaults

Open
#22,329 2 comments 1 reaction 1 assignee Claimed by @nathanb9 View on GitHub
enhancement ffi physical-plan
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Gap

`FFI_ExecutionPlan` in `datafusion/ffi/src/execution_plan.rs` does not plumb many defaulted methods of `ExecutionPlan`. Producer overrides are silently lost on the consumer side, breaking optimizer decisions that depend on them.

## Missing methods

- `required_input_distribution`
- `required_input_ordering`
- `maintains_input_order`
- `benefits_from_input_partitioning`
- `check_invariants`
- `apply_expressions`
- `reset_state`
- `with_fetch`
- `fetch`
- `supports_limit_pushdown`
- `cardinality_effect`
- `partition_statistics`
- `try_swapping_with_projection`
- `gather_filters_for_pushdown`
- `handle_child_pushdown_result`
- `with_new_state`
- `try_pushdown_sort`
- `with_preserve_order`

## Why it matters

**Severity: critical, largest single gap by method count.** Distribution / ordering requirements drive repartition + sort insertion. Filter pushdown (`gather_filters_for_pushdown` / `handle_child_pushdown_result`) is entirely disabled across the FFI boundary today. Limit pushdown, projection swapping, cardinality-based costing, and partition statistics all silently fall back to trait defaults.

## Implementation notes

- Plumb each as a plain `unsafe extern \"C\" fn`; wrapper body calls the trait method on inner `Arc` and dispatch picks override-or-default.
- Layout change → `api change` label, target `main` only, no back-port to `branch-`.
- Add unit tests (local-bypass + `mock_foreign_marker_id` forced-foreign) **and** integration tests under `datafusion/ffi/tests/` — layout change makes integration coverage mandatory.
- Consider splitting into sub-PRs by theme (pushdown / ordering / fetch) if a single PR is unwieldy, but ship them all into one major bump.

---

Generated from `datafusion-ffi` skill audit. See `.ai/skills/datafusion-ffi/SKILL.md` §"Method coverage" and §"Known gaps to close" (originated in PR #22327). If a PR addressing this finds any item to be a false positive (e.g., a method intentionally omitted for a documented reason), please also propose an update to the `datafusion-ffi` skill so future audits do not re-flag it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.