apache / apache/auron

Support MapType payloads in native round-robin shuffle

Open
#2,470 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

**Is your feature request related to a problem? Please describe.**

Auron currently falls back from native round-robin shuffle when the shuffle payload contains a Spark `MapType`, including maps nested in arrays or structs.

Native shuffle serialization already supports Map values. However, round-robin shuffle sorts its input before repartitioning to provide stable partition assignment, and Arrow row encoding does not support `MapType`.

**Describe the solution you'd like**

Allow `MapType` payloads in native round-robin shuffle.

When the input schema contains a Map, use Auron's Spark-compatible Murmur3 hash over all payload columns as the internal sort key. Keep the existing all-column sort path unchanged for schemas without maps.

Apply the same behavior to both local and RSS shuffle writers, and add coverage asserting that Map repartitioning uses `NativeShuffleExchangeExec`.

**Describe alternatives you've considered**

Skipping the pre-sort for Map payloads was considered, but that would weaken stable round-robin partition assignment.

Adding Map ordering support to Arrow row encoding would be a significantly broader change, and Map is not an orderable Spark SQL type.

**Additional context**

This only allows Map values to be carried as round-robin shuffle payloads. It does not add support for using MapType as a range sort key or hash partitioning key.

Contributor guide

Open the contributing guide

Research direction

Locate the local and RSS shuffle writers and the NativeShuffleExchangeExec coverage, then trace how round-robin shuffle chooses its sort key for schemas containing MapType, including nested maps. Done means Map payloads use Auron's Spark-compatible Murmur3 hash for stable repartitioning in both writers, while schemas without maps keep the existing all-column sort path, with tests asserting NativeShuffleExchangeExec is used.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.