Carry SedonaScalarUDF metadata across the kernel FFI (and sync the s2geography submodule's vendored header)
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
## Background
`SedonaScalarUDF` now carries a generic `metadata: HashMap` map (e.g. `NEEDS_PIXELS_METADATA_KEY`, read by the `RS_EnsureLoaded` optimizer rule). That metadata currently lives only in-process — it does **not** cross the `sedona-extension` kernel FFI, so a plugin-defined UDF can't declare planner-visible flags across the cdylib boundary.
An attempt to add it (a `metadata` callback on `SedonaCScalarKernel`) was reverted in #886 because it broke s2geography init.
## The landmine
`SedonaCScalarKernel` is defined in **two** places that must stay byte-for-byte ABI-compatible:
1. `c/sedona-extension/src/extension.rs` (the Rust `#[repr(C)]` struct) + `c/sedona-extension/src/sedona_extension.h`.
2. A **vendored copy inside the `sedona-s2geography` submodule**: `c/sedona-s2geography/s2geography/src/s2geography/sedona_udf/sedona_extension.h`.
`s2geography_c.cc::S2GeogInitKernels` lays out the struct from copy (2); `c/sedona-s2geography/src/kernels.rs` calls it with `size_of::()` from copy (1). Changing the struct in (1) only — even adding a field — shifts size/offsets and makes `S2GeogInitKernels` fail (`code: 22`), panicking every `SedonaContext` construction.
## What this task must do
- Add the `metadata` callback to `SedonaCScalarKernel` — **appending at the end of the struct, never inserting mid-struct.**
- Update the submodule's vendored `sedona_extension.h` to match, in the `sedona-s2geography` fork, and **bump the submodule pointer** here in lockstep.
- Re-add the export (`ExportedScalarKernel::with_metadata`) / import (`ImportedScalarKernel::metadata()`) plumbing + the JSON round-trip serialization (`serde_json`) + round-trip test.
- Apply imported metadata to the assembled `SedonaScalarUDF`.
## Note for the `FFI_ObjectStore` work (#890)
#890 extends this same FFI surface. Any `SedonaCScalarKernel` (or related header) change there hits the identical dual-definition constraint — the vendored submodule copy must be updated and the submodule bumped together. Coordinate.
Contributor guide
Research direction
Start in c/sedona-extension/src/extension.rs and its sedona_extension.h, then compare c/sedona-s2geography/s2geography/src/s2geography/sedona_udf/sedona_extension.h and c/sedona-s2geography/src/kernels.rs. Trace the existing ExportedScalarKernel/ImportedScalarKernel metadata and serde_json round-trip test. Done means the callback is appended, both headers and the submodule pointer match, metadata reaches SedonaScalarUDF, and the round trip passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, rust
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100