apache / apache/datafusion

FFI: `FFI_ScalarUDF` silently drops producer overrides of defaulted trait methods

Open
#22,330 4 comments 0 reactions 0 assignees View on GitHub
enhancement ffi functions
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Gap

`FFI_ScalarUDF` in `datafusion/ffi/src/udf/mod.rs` does not plumb several defaulted methods of `ScalarUDFImpl`. Producer overrides are silently lost on the consumer side.

## Missing methods

- `display_name`
- `schema_name`
- `with_updated_config`
- `simplify`
- `preimage`
- `conditional_arguments`
- `evaluate_bounds`
- `propagate_constraints`
- `struct_field_mapping`
- `output_ordering`
- `preserves_lex_ordering`
- `placement`
- `documentation`

## Why it matters

**Severity: critical.** `simplify` / `preimage` / `evaluate_bounds` / `propagate_constraints` are optimizer hooks — silent loss disables rewrites and interval reasoning across the FFI boundary. `display_name` / `schema_name` produce wrong SQL output naming on the consumer side. `placement` / `conditional_arguments` affect planning correctness for short-circuiting and column placement.

## Implementation notes

- Plumb each as a plain `unsafe extern \"C\" fn`; wrapper body calls the trait method on inner `Arc` and dispatch handles override-or-default.
- Methods that ship `Expr` (e.g. `simplify`, `preimage`) require the embedded `FFI_LogicalExtensionCodec` for serialization.
- Layout change → `api change` label, target `main` only, no back-port to `branch-`.
- Add unit tests (local-bypass + `mock_foreign_marker_id` forced-foreign) **and** integration tests under `datafusion/ffi/tests/` for any method shipping non-trivial FFI types.

---

Generated from `datafusion-ffi` skill audit. See `.ai/skills/datafusion-ffi/SKILL.md` §"Method coverage" and §"Known gaps to close" (originated in PR #22327). If a PR addressing this finds any item to be a false positive (e.g., a method intentionally omitted for a documented reason), please also propose an update to the `datafusion-ffi` skill so future audits do not re-flag it.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/ffi/src/udf/mod.rs by comparing FFI_ScalarUDF with ScalarUDFImpl and the method-coverage notes in .ai/skills/datafusion-ffi/SKILL.md. Trace the existing dispatch and FFI_LogicalExtensionCodec handling, then run the relevant local-bypass and forced-foreign tests. Done means all listed overrides are preserved, with unit tests and integration tests in datafusion/ffi/tests/ for non-trivial FFI types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.