apache / apache/datafusion-comet
`translate` falls back to Spark by default instead of using the codegen dispatcher like the other string functions
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`CometStringTranslate` reports `Incompatible` unconditionally:
```scala
override def getSupportLevel(expr: StringTranslate): SupportLevel = Incompatible(
Some(incompatReason))
```
`spark/src/main/scala/org/apache/comet/serde/strings.scala:119-120`
The reason is real — DataFusion's `translate` iterates Unicode graphemes where Spark uses code points, and substitutes U+0000 rather than treating it as a deletion sentinel. But with no `CodegenDispatchFallback` mixin, the consequence is that `translate` falls the whole projection back to Spark unless the user sets `spark.comet.expression.StringTranslate.allowIncompatible=true` and accepts wrong answers.
Every comparable string function in the same file already has the better shape: compatible by default through the dispatcher, native as an explicit opt-in. See `CometInitCap` (`:146`), `CometStringReplace` (`:177`), `CometCaseConversionBase` for `upper` / `lower` (`:50`), and the regex family at `:368`, `:470`, `:508`, `:545`, `:598`.
Fix: mix `CodegenDispatchFallback` into `CometStringTranslate`. `StringTranslate` is a plain codegen expression over `StringType`, so `canHandle` admits it.
Once that lands, `docs/source/user-guide/latest/expressions.md` should report `translate` as Hybrid, and the existing note about the grapheme/code-point difference becomes an opt-in caveat rather than a description of default behavior.
Contributor guide
Assessment
This issue has not been assessed yet.