apache / apache/datafusion-comet
Support NullType output types in codegen dispatch
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Any expression whose output type contains `NullType` is rejected by the codegen dispatch gate and the whole operator falls back to Spark, even though a `NullType` column can only ever hold nulls.
Spark's untyped constructors leave `NullType` children behind, so this hits common literals:
- `map()` → `MapType(NullType, NullType)`
- `map('a', NULL)` → `MapType(StringType, NullType)`
- `array()` → `ArrayType(NullType)`
- `map_from_arrays(array(), array())`, `struct(map())`, and anything nested around them
`CreateMap` and friends are routed through codegen dispatch (`spark/src/main/scala/org/apache/comet/serde/maps.scala:214`), so a projection containing one of these literals is currently kicked out of the native plan with `codegen dispatch: unsupported output type ...`.
Why the gate rejects it today:
- `isSupportedDataType` (`spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala:85`) has no `NullType` case, so it falls through to `false`, recursively for `ArrayType` / `StructType` / `MapType` children.
- `canHandle` applies that same predicate to both the output type (`CometBatchKernelCodegen.scala:120`) and every `BoundReference` input (`CometBatchKernelCodegen.scala:172`).
The rejection is only needed on the input side: `CometScalaUDFCodegen.specFor` has no way to build an `ArrowColumnSpec` for a `NullVector`, so a `NullType` *input* must keep falling back. On the output side the kernel only needs to emit an all-null Arrow `NullVector`, which the rest of the pipeline already understands (`serializeDataType` maps `NullType` to its own type id and `Utils.toArrowField` maps it to `ArrowType.Null`).
### Describe the potential solution
Make the type gate asymmetric:
- Accept `NullType` (top-level or nested inside array / struct / map) for the **output** type in `canHandle`, and keep rejecting it for `BoundReference` **inputs**.
- Teach the output emitter (`CometBatchKernelCodegenOutput`) to map `NullType` to `NullVector` and to write it with `setNull` only, without reading a source value.
- Keep the output emitter's type surface in sync with the gate, as the existing doc comments require, so plan-time acceptance never turns into an execute-time exception.
- Update the Scala/Java UDF user guide: `NullType` arguments remain unsupported, `NullType` return types become supported.
Done when a `CometSqlFileTestSuite` fixture such as `SELECT map()`, `SELECT map('a', NULL)`, `SELECT array(map())`, and a `map()` column carried through `ORDER BY` runs without a Spark fallback, and unit tests in `CometCodegenSourceSuite` lock in the output/input asymmetry and the gate/emitter agreement.
### Additional context
No native changes are needed: the `NullType` support in serde and Arrow field conversion already exists; the missing piece is entirely in the JVM codegen gate and output emitter.
I have a patch for this ready and will open a PR referencing this issue.
Assisted-by: Claude Code (claude-fable-5)
Contributor guide
Research direction
Start with isSupportedDataType and canHandle in spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala, then inspect CometBatchKernelCodegenOutput and CometScalaUDFCodegen.specFor. Read the Scala/Java UDF user guide and run the CometCodegenSourceSuite plus a CometSqlFileTestSuite fixture. Done means supported NullType outputs execute without Spark fallback while NullType inputs remain unsupported, with the gate and emitter agreeing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100