apache / apache/datafusion-comet
`length` / `bit_length` / `octet_length` fall back to Spark on binary input
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`length`, `bit_length` and `octet_length` all decline `BinaryType` input, and each falls the whole projection back to Spark:
```scala
case _: BinaryType => Unsupported(Some("Length on BinaryType is not supported"))
```
`spark/src/main/scala/org/apache/comet/serde/strings.scala:89` (`CometLength`), `:98` (`CometBitLength`), `:107` (`CometOctetLength`)
All three are trivial in Spark's generated code — `length(binary)` is `.numBytes()`, and the other two are arithmetic on it. `BinaryType` is in `CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher handles them.
Fix: mix `CodegenDispatchFallback` into all three. They are near-identical serdes, so this is one small change.
The generated docs list all three as "✅ Native" with no note today, so they should pick up the Hybrid marker once the mixin is added.
Contributor guide
Assessment
This issue has not been assessed yet.