Align Comet's native UDF surface with SPARK-55278's language-agnostic UDF protocol

## Background

[SPARK-55278: Language-agnostic UDF Protocol for Spark](https://issues.apache.org/jira/browse/SPARK-55278) is an accepted SPIP, currently under implementation and scoped at Spark 4.3.0/4.4.0. It defines a unified API and execution protocol for user code that runs outside the Spark engine process, standardizing on Arrow for data, gRPC for the service definition, a declarative worker specification, and language-agnostic Catalyst logical and physical nodes so the planner can reason about an external UDF without knowing what language implements it.

This was raised by @wForget on #4459, which adds Comet's own native scalar UDF path. That PR should not wait on this, but the two surfaces will have to be reconciled, and it is cheaper to know where they diverge before Comet's API has users.

## Where the two designs actually differ

Comet's path and the SPIP solve overlapping problems with opposite process models, so this is not a matter of adopting one wholesale.

| | Comet native UDFs (#4459) | SPARK-55278 |
| --- | --- | --- |
| Locality | in-process, `dlopen` of a cdylib in the executor | out-of-process worker, IPC |
| Data | Arrow C Data Interface, zero copy | Arrow over gRPC |
| Coupling | C ABI of function pointers, versioned per Comet release | gRPC service definition, versioned by Spark |
| Planning | Comet serde emits `NativeScalarUdf` in its own proto | language-agnostic Catalyst nodes |
| Failure domain | a UDF bug can take down the executor | worker crash is isolated |

The in-process design is the reason Comet's path exists: it puts user code inside the DataFusion plan with no serialization boundary, which is exactly what an out-of-process protocol cannot offer. So the SPIP does not replace it.

## What alignment could mean

Worth deciding, roughly in order of how much it constrains #4459's API:

1. **Recognize the SPIP's Catalyst nodes.** Once Spark ships language-agnostic logical/physical UDF nodes, Comet's `CometExecRule` and `QueryPlanSerde` will see them in plans whether or not Comet participates. At minimum Comet needs a deliberate fallback story for them; at best a Comet-native execution of a worker-backed UDF.
2. **Reuse the SPIP's registration vocabulary.** `CometNativeUDF.register` currently invents its own signature declaration (name, library path, input types, return type, determinism). If the SPIP's worker specification names the same concepts, Comet should borrow the names and shapes rather than diverge for no reason. This is the cheapest thing to get right and the most annoying to change later.
3. **Serve a SPIP worker from a Comet-native plan.** A worker-backed UDF inside a `CometNativeExec` would otherwise force a transition back to the JVM. Whether Comet can drive the protocol from Rust is an open question and probably the largest piece here.
4. **Decide whether the C ABI stays.** If the SPIP's protocol becomes the way non-JVM UDFs reach Spark, Comet's C ABI is either a fast path for the in-process case that the protocol cannot cover, or a duplicate surface to deprecate. The versioning argument recorded in `comet-udf-sdk` (no DataFusion type crosses the boundary, so a compiled UDF survives Comet's DataFusion upgrades) applies to the C ABI and not obviously to a gRPC protocol Spark versions on its own cadence.

## Not blocking

Comet supports Spark 3.4 through 4.1 today, so anything landing in Spark 4.3+ is some way off, and the SPIP's own subtasks are still in progress. This issue is to track the design and to keep #4459's follow-ups from painting Comet into a corner, not to gate that PR.

## References

- SPARK-55278: https://issues.apache.org/jira/browse/SPARK-55278
- Comet PR #4459 and the review thread where this was raised
- Comet issue #747, the original request for custom native UDFs
- Comet issue #5301, publishing a C header for the current ABI

Start by reading Comet PR #4459 and its review thread, then inspect CometExecRule, QueryPlanSerde, and the comet-udf-sdk versioning discussion. Compare those surfaces with SPARK-55278's Catalyst nodes, worker specification, and gRPC protocol. Done means documenting or implementing an agreed alignment and fallback strategy without blocking #4459.

Written by the indexing model from the issue text.