apache / apache/datafusion-comet

`named_struct` with duplicate field names falls back to Spark

Open
#5,586 1 comment 0 reactions 1 assignee Claimed by @RRXXZZYY View on GitHub
area:expressions enhancement good first issue
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`named_struct` with duplicate field names falls the whole projection back to Spark:

```scala
if (expr.names.length != expr.names.distinct.length) {
Unsupported(Some(duplicateNamesReason))
}
```

`spark/src/main/scala/org/apache/comet/serde/structs.scala:42-43`

`docs/source/user-guide/latest/expressions.md` records it as "Duplicate field names fall back".

Spark permits duplicate field names in a struct and `CreateNamedStruct.doGenCode` builds the row positionally, so the dispatcher reproduces it exactly. `StructType` is supported by `CometBatchKernelCodegen.isSupportedDataType` recursively, and duplicate names are not a problem on the Arrow side — the kernel's output field construction is positional too. Worth confirming with a test that the resulting Arrow schema round-trips, since duplicate field names in an Arrow struct are legal but unusual.

Fix: mix `CodegenDispatchFallback` into `CometCreateNamedStruct`.

`CometCreateNamedStruct` also declines inside `convert` at `:66` when a child fails to serialize, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.