apache / apache/datafusion-comet

[Feature] support concat() for BinaryType and ArrayType inputs

Open
#4,471 2 comments 0 reactions 0 assignees View on GitHub
area:expressions enhancement priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

## Describe the bug

Spark's `concat(...)` accepts `StringType`, `BinaryType`, and `ArrayType` arguments (`Concat.allowedTypes = Seq(StringType, BinaryType, ArrayType)` in `collectionOperations.scala`, widened to `StringTypeWithCollation` in Spark 4.0+). Comet's `CometConcat` only natively supports `StringType` children; for `BinaryType` or `ArrayType` it falls back to Spark.

Surfaced by the collection-expressions audit in the collection-expressions audit PR. The audit relabels the `getSupportLevel` branch from `Incompatible` to `Unsupported` (the fallback is a genuine "Comet does not support" case, not a wrong-result case), but the underlying coverage gap remains.

## Steps to reproduce

```sql
-- BinaryType
SELECT concat(unhex('CAFE'), unhex('BEEF'));

-- ArrayType
CREATE TABLE t(a array, b array) USING parquet;
INSERT INTO t VALUES (array(1, 2), array(3, 4));
SELECT concat(a, b) FROM t;
```

Both queries currently fall back to Spark.

## Expected behavior

Native support for `concat` over `BinaryType` (concatenate byte arrays) and `ArrayType` (concatenate arrays, equivalent to `array_concat`).

## Additional context

- Serde: `CometConcat` in `spark/src/main/scala/org/apache/comet/serde/strings.scala`
- DataFusion has `array_concat` for the array case; Comet already wires it for the array_concat function. The work is to route `Concat(...)` through the same native path.
- For BinaryType, DataFusion's `concat` UDF is Utf8-only, so a Comet-side helper or upstream patch would be needed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.