apache / apache/datafusion-comet

String expression (collation) audit followup

Open
#4,573 1 comment 1 reaction 0 assignees View on GitHub
area:expressions bug priority:low
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### Describe the bug

## Expressions Missing Collation Guards

Tracked in [#2190](https://github.com/apache/datafusion-comet/issues/2190). Currently unreachable because the scan-level guard (`CometScanRule.scala:731`) forces full fallback for any collated column. These become reachable if the scan guard is relaxed.

### Collation-sensitive (need guards)

- [ ] `Upper` / `Lower` — `strings.scala:54-76` (partially mitigated by `COMET_CASE_CONVERSION_ENABLED` gate)
- [ ] `InitCap` — `strings.scala:87`
- [ ] `Like` — `strings.scala:246`
- [ ] `RLike` — `strings.scala:264`
- [ ] `ConcatWs` — `strings.scala:225`
- [ ] `Contains` — `QueryPlanSerde.scala:165` (generic `CometScalarFunction`)
- [ ] `StartsWith` — `QueryPlanSerde.scala:176` (generic `CometScalarFunction`)
- [ ] `EndsWith` — `QueryPlanSerde.scala:166` (generic `CometScalarFunction`)
- [ ] `StringInstr` — `QueryPlanSerde.scala:177` (generic `CometScalarFunction`)
- [ ] `StringReplace` — `QueryPlanSerde.scala:179` (generic `CometScalarFunction`)
- [ ] `ArrayJoin` — `arrays.scala:377`

### Likely safe (verify before closing)

- [ ] `StringRepeat` — `strings.scala:34` (repeating bytes preserves collation)
- [ ] `Substring` / `Left` / `Right` — `strings.scala:106,132,166` (byte slicing)
- [ ] `StringLPad` / `StringRPad` — `strings.scala:296,325` (padding, but comparison unaffected)
- [ ] `RegExpReplace` — `strings.scala:353` (regex on raw bytes)
- [ ] `StringSplit` — `strings.scala:402` (splitting on raw bytes)
- [ ] `GetJsonObject` — `strings.scala:428` (JSON extraction)

### Safe (no guard needed)

- [x] `Length` — `strings.scala:78` (char count independent of collation)
- [x] `Ascii` — `QueryPlanSerde.scala` (first byte value)
- [x] `BitLength` / `OctetLength` — `QueryPlanSerde.scala` (byte-level metric)
- [x] `Chr` — `QueryPlanSerde.scala` (int → char, no comparison)

### Already guarded

- [x] `Concat` — `strings.scala` ([#4567](https://github.com/apache/datafusion-comet/pull/4567))
- [x] `Reverse` — `collectionOperations.scala` ([#4567](https://github.com/apache/datafusion-comet/pull/4567))
- [x] `ArrayIntersect` — `arrays.scala:208`

### Steps to reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.