apache / apache/datafusion-comet
`named_struct` with duplicate field names falls back to Spark
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`named_struct` with duplicate field names falls the whole projection back to Spark:
```scala
if (expr.names.length != expr.names.distinct.length) {
Unsupported(Some(duplicateNamesReason))
}
```
`spark/src/main/scala/org/apache/comet/serde/structs.scala:42-43`
`docs/source/user-guide/latest/expressions.md` records it as "Duplicate field names fall back".
Spark permits duplicate field names in a struct and `CreateNamedStruct.doGenCode` builds the row positionally, so the dispatcher reproduces it exactly. `StructType` is supported by `CometBatchKernelCodegen.isSupportedDataType` recursively, and duplicate names are not a problem on the Arrow side — the kernel's output field construction is positional too. Worth confirming with a test that the resulting Arrow schema round-trips, since duplicate field names in an Arrow struct are legal but unusual.
Fix: mix `CodegenDispatchFallback` into `CometCreateNamedStruct`.
`CometCreateNamedStruct` also declines inside `convert` at `:66` when a child fails to serialize, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines.
Contributor guide
Assessment
This issue has not been assessed yet.