Support MapType payloads in native round-robin shuffle
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
**Is your feature request related to a problem? Please describe.**
Auron currently falls back from native round-robin shuffle when the shuffle payload contains a Spark `MapType`, including maps nested in arrays or structs.
Native shuffle serialization already supports Map values. However, round-robin shuffle sorts its input before repartitioning to provide stable partition assignment, and Arrow row encoding does not support `MapType`.
**Describe the solution you'd like**
Allow `MapType` payloads in native round-robin shuffle.
When the input schema contains a Map, use Auron's Spark-compatible Murmur3 hash over all payload columns as the internal sort key. Keep the existing all-column sort path unchanged for schemas without maps.
Apply the same behavior to both local and RSS shuffle writers, and add coverage asserting that Map repartitioning uses `NativeShuffleExchangeExec`.
**Describe alternatives you've considered**
Skipping the pre-sort for Map payloads was considered, but that would weaken stable round-robin partition assignment.
Adding Map ordering support to Arrow row encoding would be a significantly broader change, and Map is not an orderable Spark SQL type.
**Additional context**
This only allows Map values to be carried as round-robin shuffle payloads. It does not add support for using MapType as a range sort key or hash partitioning key.
Contributor guide
Research direction
Locate the local and RSS shuffle writers and the NativeShuffleExchangeExec coverage, then trace how round-robin shuffle chooses its sort key for schemas containing MapType, including nested maps. Done means Map payloads use Auron's Spark-compatible Murmur3 hash for stable repartitioning in both writers, while schemas without maps keep the existing all-column sort path, with tests asserting NativeShuffleExchangeExec is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100