apache / apache/datafusion-comet
Reduce per-row sorting allocations in multi-entry map_sort normalization
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Normalizing multi-entry `map` batches currently slices each row's keys and invokes Arrow sorting, allocating temporary indices and prefix tuples for every map. The singleton optimization in #5887 avoids this work for one-entry maps, but multi-entry maps still pay these costs.
Reuse an Arrow-compatible prefix-tuple scratch buffer across rows for multi-entry string-key/int-value maps, while preserving the singleton fast path, exact duplicate-key permutations, sliced offsets, null buffers, schema metadata, and unsupported-key errors. Also fill rebased offsets in bulk for all-empty visible batches.
Matched release microbenchmarks show about 3x faster normalization for 2–10-entry forward maps and 33–39% faster normalization for 2–50-entry maps in independent paired measurements. Validation covers normalization, hashing, combined execution, nulls, mixed cardinalities, Unicode keys, and fallback types.
Related to #5818 and the matched benchmark work in #5822.
Contributor guide
Research direction
No file or test path is named. Read the singleton optimization in #5887 and the related work in #5818 and #5822, then trace multi-entry map_sort normalization and its validation coverage. Done means the listed normalization, hashing, execution, null, Unicode, fallback, and benchmark behavior remains correct with the reported allocation and speed improvements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100