apache / apache/datafusion-comet

cast_map_to_map drops entries null buffer and ignores target sorted flag

Open
#5,097 1 comment 0 reactions 1 assignee Claimed by @Smallfu666 View on GitHub
area:expressions bug priority:low
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

`cast_map_to_map` in `native/spark-expr/src/conversion_funcs/cast.rs` rebuilds the map's keys, values, entries struct, and offsets by hand, mainly to reconcile Parquet field naming ("key_value") with Spark naming ("entries"). In doing so it diverges from arrow's map cast in two ways:

1. It passes `None` for the entries-struct null buffer, dropping `map_array.entries().nulls()` if present.
2. It always propagates the source's `sorted` flag (`*from_sorted`) and ignores the target's `_to_sorted`.

Arrow's Map-to-Map `cast_with_options` preserves both correctly.

### Steps to reproduce

Latent today: Spark-produced maps do not carry null map entries, so the dropped null buffer is not currently observable through normal plans. The incorrect `sorted` flag is metadata-only.

### Expected behavior

For the rename-only case (`from_key_type == to_key_type && from_value_type == to_value_type`, the common Parquet case), delegate to arrow's map cast (or the existing zero-copy `relabel_array` in `core/src/parquet/cast_column.rs`), which handles both issues. Keep the hand-rolled recursion only when key or value types genuinely change and need Spark cast semantics.

### Additional context

Found during an audit of native code that replicates existing arrow-rs kernels. Filed as a bug because the current code drops data (a null buffer) relative to its input, even though no user-visible query is known to be affected yet.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.