apache / apache/datafusion

FFI: `FFI_PartitionEvaluator` missing `memoize`

Open
#22,336 1 comment 0 reactions 1 assignee Claimed by @sandugood View on GitHub
enhancement ffi functions
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Gap

`FFI_PartitionEvaluator` in `datafusion/ffi/src/udwf/partition_evaluator.rs` does not plumb `PartitionEvaluator::memoize`. Producers that override `memoize` to cache work between consecutive window evaluations have the override silently discarded.

## Missing method

- `memoize`

## Why it matters

**Severity: minor.** Performance regression only — correctness unaffected. But for window functions that rely on memoization (e.g. ranking variants that reuse prior-frame state), the consumer side recomputes from scratch every frame.

## Implementation notes

- Plumb as `unsafe extern \"C\" fn`; wrapper body calls `inner.memoize(...)`.
- Layout change → `api change` label, target `main` only, no back-port to `branch-`.
- Add unit tests (local-bypass + `mock_foreign_marker_id` forced-foreign). Integration test under `datafusion/ffi/tests/` if memoization state crosses the boundary.

---

Generated from `datafusion-ffi` skill audit. See `.ai/skills/datafusion-ffi/SKILL.md` §"Method coverage" and §"Known gaps to close" (originated in PR #22327). If a PR addressing this finds the omission to be a false positive, please also propose an update to the `datafusion-ffi` skill so future audits do not re-flag it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.