apache / apache/datafusion-comet
cast_map_to_map drops entries null buffer and ignores target sorted flag
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
`cast_map_to_map` in `native/spark-expr/src/conversion_funcs/cast.rs` rebuilds the map's keys, values, entries struct, and offsets by hand, mainly to reconcile Parquet field naming ("key_value") with Spark naming ("entries"). In doing so it diverges from arrow's map cast in two ways:
1. It passes `None` for the entries-struct null buffer, dropping `map_array.entries().nulls()` if present.
2. It always propagates the source's `sorted` flag (`*from_sorted`) and ignores the target's `_to_sorted`.
Arrow's Map-to-Map `cast_with_options` preserves both correctly.
### Steps to reproduce
Latent today: Spark-produced maps do not carry null map entries, so the dropped null buffer is not currently observable through normal plans. The incorrect `sorted` flag is metadata-only.
### Expected behavior
For the rename-only case (`from_key_type == to_key_type && from_value_type == to_value_type`, the common Parquet case), delegate to arrow's map cast (or the existing zero-copy `relabel_array` in `core/src/parquet/cast_column.rs`), which handles both issues. Keep the hand-rolled recursion only when key or value types genuinely change and need Spark cast semantics.
### Additional context
Found during an audit of native code that replicates existing arrow-rs kernels. Filed as a bug because the current code drops data (a null buffer) relative to its input, even though no user-visible query is known to be affected yet.
Contributor guide
Assessment
This issue has not been assessed yet.