apache / apache/datafusion-comet

Reduce per-row sorting allocations in multi-entry map_sort normalization

Closed
#5,900 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Normalizing multi-entry `map` batches currently slices each row's keys and invokes Arrow sorting, allocating temporary indices and prefix tuples for every map. The singleton optimization in #5887 avoids this work for one-entry maps, but multi-entry maps still pay these costs.

Reuse an Arrow-compatible prefix-tuple scratch buffer across rows for multi-entry string-key/int-value maps, while preserving the singleton fast path, exact duplicate-key permutations, sliced offsets, null buffers, schema metadata, and unsupported-key errors. Also fill rebased offsets in bulk for all-empty visible batches.

Matched release microbenchmarks show about 3x faster normalization for 2–10-entry forward maps and 33–39% faster normalization for 2–50-entry maps in independent paired measurements. Validation covers normalization, hashing, combined execution, nulls, mixed cardinalities, Unicode keys, and fallback types.

Related to #5818 and the matched benchmark work in #5822.

Contributor guide

Open the contributing guide

Research direction

No file or test path is named. Read the singleton optimization in #5887 and the related work in #5818 and #5822, then trace multi-entry map_sort normalization and its validation coverage. Done means the listed normalization, hashing, execution, null, Unicode, fallback, and benchmark behavior remains correct with the reported allocation and speed improvements.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.