apache / apache/datafusion-comet
`to_csv` never runs inside Comet by default, unlike `to_json` / `from_csv` / `schema_of_csv`
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`to_csv` never runs inside Comet by default. Every path through `CometStructsToCsv.getSupportLevel` returns `Unsupported` or `Incompatible`:
```scala
if (containsComplexType) {
return Unsupported(...)
}
if (containsIncompatibleDataTypes) {
return Incompatible(...)
}
// https://github.com/apache/datafusion-comet/issues/3232
Incompatible()
```
`spark/src/main/scala/org/apache/comet/serde/structs.scala:271-288`
The serde has no `CodegenDispatchFallback` mixin, so with the default configuration the enclosing operator falls back to Spark for every `to_csv` call. The native path is reachable only with `spark.comet.expression.StructsToCsv.allowIncompatible=true`.
The rest of the structured-text family does not behave this way. `to_json` is Hybrid, and `from_csv` and `schema_of_csv` are both plain `CometCodegenDispatch` (`serde/csv.scala:24`, `:26`). `to_csv` is the odd one out.
There is also a docs bug here: `docs/source/user-guide/latest/expressions.md` lists `to_csv` as "✅ Native" with no note, which reads as "runs natively by default". It does not. Adding the mixin fixes the docs too, since `GenerateDocs` derives the Implementation column from the trait mixins and would then report it as Hybrid.
Fix: mix `CodegenDispatchFallback` into `CometStructsToCsv`. The complex-type arm dispatches fine — arrays, structs and maps are all in `CometBatchKernelCodegen.isSupportedDataType` as long as their leaves are.
Contributor guide
Assessment
This issue has not been assessed yet.