apache / apache/datafusion-comet

`length` / `bit_length` / `octet_length` fall back to Spark on binary input

Open
#5,584 1 comment 0 reactions 1 assignee Claimed by @adibmbrk View on GitHub
area:expressions enhancement good first issue
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`length`, `bit_length` and `octet_length` all decline `BinaryType` input, and each falls the whole projection back to Spark:

```scala
case _: BinaryType => Unsupported(Some("Length on BinaryType is not supported"))
```

`spark/src/main/scala/org/apache/comet/serde/strings.scala:89` (`CometLength`), `:98` (`CometBitLength`), `:107` (`CometOctetLength`)

All three are trivial in Spark's generated code — `length(binary)` is `.numBytes()`, and the other two are arithmetic on it. `BinaryType` is in `CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher handles them.

Fix: mix `CodegenDispatchFallback` into all three. They are near-identical serdes, so this is one small change.

The generated docs list all three as "✅ Native" with no note today, so they should pick up the Hybrid marker once the mixin is added.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.