apache / apache/datafusion-comet

Tighten CometUDF API with input/return type validation at registration

Open
#4,173 0 comments 0 reactions 0 assignees View on GitHub
area:expressions priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Describe the problem

The `CometUDF` trait (`common/src/main/scala/org/apache/comet/udf/CometUDF.scala`, introduced on the JVM-scalar-UDF prototype branch `prototype-jvm-scalar-udf`) is too thin to support fail-fast validation:

```scala
trait CometUDF {
def evaluate(inputs: Array[ValueVector]): ValueVector
}
```

Today the proto carries `class_name`, `args`, `return_type`, and `return_nullable`. There is no compile-time or registration-time check that the UDF actually consumes the declared input types or produces the declared return type. A mismatch (e.g. UDF returns `IntVector` but proto declares Boolean) only manifests as a hard JVM crash inside Arrow's `from_ffi` on the native side.

## Describe the potential solution

Extend the trait with self-describing methods:

```scala
trait CometUDF {
def inputTypes: Seq[DataType]
def returnType: DataType
def nullable: Boolean
def evaluate(inputs: Array[ValueVector]): ValueVector
}
```

The bridge can then validate `inputs.length == inputTypes.length` and that each input vector's Arrow type matches `inputTypes(i)` before dispatch, and that the returned vector matches `returnType` before export. Mismatches produce a Java exception that surfaces as a clean DataFusion error rather than crashing the executor.

A registry could also be added so users register `CometUDF` impls by name, decoupling the convert path from `Class.forName` lookups and making the API's surface explicit.

## Additional context

Identified during code review of the JVM-scalar-UDF prototype. Filed for follow-up so the prototype PR can ship without a public-API design decision.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.