apache / apache/datafusion-comet
Derive a native UDF's return type from the library instead of requiring the caller to declare it
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
## Problem
`CometNativeUDF.register` asks the caller for the full signature:
```scala
CometNativeUDF.register(spark, "add_one_c", libPath, Seq(LongType), LongType)
```
but the library already knows most of it. A kernel's `return_field` computes the output type from the argument types, and Comet calls it at planning time to check the declaration, so the declared `returnType` is verified against the library rather than being the source of truth. Raised by @wForget on #4459: can `inputTypes` / `returnType` / `deterministic` come from the library instead?
## Where each one stands
**`returnType`** is the strongest candidate, and it is already redundant in the sense that a wrong value is rejected. The obstacle is ordering, not information: Spark's analyzer needs a concrete `DataType` when `register` installs the catalog stub, and at that moment the argument types of the eventual call sites are not known. Deriving it would mean calling the library on the driver with the `inputTypes` the caller supplied, which removes one of the two declarations but not both.
**`inputTypes`** cannot be derived as things stand. The ABI has no way to enumerate a kernel's accepted signatures: `return_field` is a predicate over argument types, not a description of them, and a kernel like `echo_c` accepts every type. Getting this from the library would need a new ABI entry point returning declared signatures, which is an ABI break and worth its own design.
**`deterministic`** is per-registration today and always has to be `true` (#5249). It is genuinely a property of the kernel rather than the registration, so if anything it belongs on the library side, but that is blocked behind honoring it at all.
## Possible shape
An overload that takes only the arguments and derives the return type:
```scala
CometNativeUDF.register(spark, "add_one_c", libPath, Seq(LongType))
```
calling the library on the driver to resolve the return type, and failing with the kernel's own message if the kernel rejects those argument types. The existing overload stays for callers who want the declaration checked.
That leaves the plan-time check in place. It is still worth keeping even when the type was derived, because the derivation happens on the driver and the check runs on the executor against the library actually loaded there, which need not be the same file.
## Related
- #4459, where this was raised
- #5249, honoring `deterministic`
- #4173, the equivalent validation gap on the JVM `CometUDF` path
Contributor guide
Research direction
Start at CometNativeUDF.register and trace the catalog-stub setup and plan-time validation, then inspect the native library's return_field path. Review the related discussions in #4459, #5249, and #4173 before assessing the proposed overload. Done means the return type can be resolved from supplied input types while retaining the existing declaration check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100