apache / apache/datafusion-comet
String expression (collation) audit followup
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### Describe the bug
## Expressions Missing Collation Guards
Tracked in [#2190](https://github.com/apache/datafusion-comet/issues/2190). Currently unreachable because the scan-level guard (`CometScanRule.scala:731`) forces full fallback for any collated column. These become reachable if the scan guard is relaxed.
### Collation-sensitive (need guards)
- [ ] `Upper` / `Lower` — `strings.scala:54-76` (partially mitigated by `COMET_CASE_CONVERSION_ENABLED` gate)
- [ ] `InitCap` — `strings.scala:87`
- [ ] `Like` — `strings.scala:246`
- [ ] `RLike` — `strings.scala:264`
- [ ] `ConcatWs` — `strings.scala:225`
- [ ] `Contains` — `QueryPlanSerde.scala:165` (generic `CometScalarFunction`)
- [ ] `StartsWith` — `QueryPlanSerde.scala:176` (generic `CometScalarFunction`)
- [ ] `EndsWith` — `QueryPlanSerde.scala:166` (generic `CometScalarFunction`)
- [ ] `StringInstr` — `QueryPlanSerde.scala:177` (generic `CometScalarFunction`)
- [ ] `StringReplace` — `QueryPlanSerde.scala:179` (generic `CometScalarFunction`)
- [ ] `ArrayJoin` — `arrays.scala:377`
### Likely safe (verify before closing)
- [ ] `StringRepeat` — `strings.scala:34` (repeating bytes preserves collation)
- [ ] `Substring` / `Left` / `Right` — `strings.scala:106,132,166` (byte slicing)
- [ ] `StringLPad` / `StringRPad` — `strings.scala:296,325` (padding, but comparison unaffected)
- [ ] `RegExpReplace` — `strings.scala:353` (regex on raw bytes)
- [ ] `StringSplit` — `strings.scala:402` (splitting on raw bytes)
- [ ] `GetJsonObject` — `strings.scala:428` (JSON extraction)
### Safe (no guard needed)
- [x] `Length` — `strings.scala:78` (char count independent of collation)
- [x] `Ascii` — `QueryPlanSerde.scala` (first byte value)
- [x] `BitLength` / `OctetLength` — `QueryPlanSerde.scala` (byte-level metric)
- [x] `Chr` — `QueryPlanSerde.scala` (int → char, no comparison)
### Already guarded
- [x] `Concat` — `strings.scala` ([#4567](https://github.com/apache/datafusion-comet/pull/4567))
- [x] `Reverse` — `collectionOperations.scala` ([#4567](https://github.com/apache/datafusion-comet/pull/4567))
- [x] `ArrayIntersect` — `arrays.scala:208`
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.