apache / apache/datafusion-comet
width_bucket bypasses CometExpressionSerde framework
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
## Describe the bug
`width_bucket` (Spark 3.5+) is wired through `CometExprShim.versionSpecificExprToProtoInternal` rather than through a `CometExpressionSerde` registered in `QueryPlanSerde.exprSerdeMap`. As a result:
- It bypasses the normal `getSupportLevel` / `getUnsupportedReasons` / `getIncompatibleReasons` hooks, so it cannot signal any incompat or unsupported branches.
- It is invisible to the auto-generated compatibility doc (`docs/source/user-guide/compatibility.md`).
- It is invisible to the per-expression `spark.comet.expression..{enabled,allowIncompatible}` configs.
- The wiring is duplicated across four shim files (`spark-3.5`, `spark-4.0`, `spark-4.1`, `spark-4.2`), so any future change has to be applied four times.
`width_bucket` also supports Spark's `YearMonthIntervalType` and `DayTimeIntervalType`, but Comet's tests only cover `DoubleType`. The native `SparkWidthBucket` declares the interval signatures, but the wiring gap means there is no way to mark them as `Unsupported` if a future bug is found.
Surfaced by the math-expressions audit (collection PR queue).
## Expected behavior
Move `width_bucket` to a `CometExpressionSerde[WidthBucket]` registered in `QueryPlanSerde.mathExpressions`, matching the pattern used by every other math expression. The serde can either accept all types and forward to the native UDF, or branch on input types and call `Unsupported` for unsupported cases.
## Additional context
- Shim location: `spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala` (plus `spark-4.0`, `spark-4.1`, `spark-4.2`)
- Native UDF: datafusion-spark `SparkWidthBucket`, registered in `native/core/src/execution/jni_api.rs`
- `width_bucket` is unsupported on Spark 3.4.3 (the function was added in 3.5).
Contributor guide
Assessment
This issue has not been assessed yet.