apache / apache/datafusion-comet

`to_csv` never runs inside Comet by default, unlike `to_json` / `from_csv` / `schema_of_csv`

Open
#5,578 1 comment 0 reactions 1 assignee Claimed by @sam-1112 View on GitHub
area:expressions documentation enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`to_csv` never runs inside Comet by default. Every path through `CometStructsToCsv.getSupportLevel` returns `Unsupported` or `Incompatible`:

```scala
if (containsComplexType) {
return Unsupported(...)
}
if (containsIncompatibleDataTypes) {
return Incompatible(...)
}
// https://github.com/apache/datafusion-comet/issues/3232
Incompatible()
```

`spark/src/main/scala/org/apache/comet/serde/structs.scala:271-288`

The serde has no `CodegenDispatchFallback` mixin, so with the default configuration the enclosing operator falls back to Spark for every `to_csv` call. The native path is reachable only with `spark.comet.expression.StructsToCsv.allowIncompatible=true`.

The rest of the structured-text family does not behave this way. `to_json` is Hybrid, and `from_csv` and `schema_of_csv` are both plain `CometCodegenDispatch` (`serde/csv.scala:24`, `:26`). `to_csv` is the odd one out.

There is also a docs bug here: `docs/source/user-guide/latest/expressions.md` lists `to_csv` as "✅ Native" with no note, which reads as "runs natively by default". It does not. Adding the mixin fixes the docs too, since `GenerateDocs` derives the Implementation column from the trait mixins and would then report it as Hybrid.

Fix: mix `CodegenDispatchFallback` into `CometStructsToCsv`. The complex-type arm dispatches fine — arrays, structs and maps are all in `CometBatchKernelCodegen.isSupportedDataType` as long as their leaves are.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.