apache / apache/datafusion-comet

`translate` falls back to Spark by default instead of using the codegen dispatcher like the other string functions

Open
#5,585 1 comment 0 reactions 1 assignee Claimed by @adibmbrk View on GitHub
area:expressions enhancement good first issue
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`CometStringTranslate` reports `Incompatible` unconditionally:

```scala
override def getSupportLevel(expr: StringTranslate): SupportLevel = Incompatible(
Some(incompatReason))
```

`spark/src/main/scala/org/apache/comet/serde/strings.scala:119-120`

The reason is real — DataFusion's `translate` iterates Unicode graphemes where Spark uses code points, and substitutes U+0000 rather than treating it as a deletion sentinel. But with no `CodegenDispatchFallback` mixin, the consequence is that `translate` falls the whole projection back to Spark unless the user sets `spark.comet.expression.StringTranslate.allowIncompatible=true` and accepts wrong answers.

Every comparable string function in the same file already has the better shape: compatible by default through the dispatcher, native as an explicit opt-in. See `CometInitCap` (`:146`), `CometStringReplace` (`:177`), `CometCaseConversionBase` for `upper` / `lower` (`:50`), and the regex family at `:368`, `:470`, `:508`, `:545`, `:598`.

Fix: mix `CodegenDispatchFallback` into `CometStringTranslate`. `StringTranslate` is a plain codegen expression over `StringType`, so `canHandle` admits it.

Once that lands, `docs/source/user-guide/latest/expressions.md` should report `translate` as Hybrid, and the existing note about the grapheme/code-point difference becomes an opt-in caveat rather than a description of default behavior.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.