apache / apache/pinot

[MSE] Support sender-side sorting for sort exchanges

Open
#19,395 0 comments 0 reactions 0 assignees View on GitHub
multi-stage query
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

## Problem

`PinotLogicalSortExchange` currently sorts only on the receiver. Operators such as WINDOW and MATCH_RECOGNIZE therefore require the receiver to buffer and sort the complete exchanged input instead of merging already sorted sender streams.

## Proposed scope

- Add sender-side sorting using the exchange collation, including direction and null ordering.
- Merge sorted mailbox streams incrementally on the receiver with bounded memory.
- Preserve cancellation, deadline, backpressure, and error propagation behavior.
- Add distributed tests covering multiple senders, duplicate keys, nulls, and empty streams.

## Context

Follow-up to #19311 and the existing receiver-side-only note in `PinotWindowExchangeNodeInsertRule`.

Contributor guide

Open the contributing guide

Research direction

Start by reading PinotLogicalSortExchange and the receiver-side-only note in PinotWindowExchangeNodeInsertRule, then review follow-up issue #19311. Trace the existing exchange and mailbox behavior before defining the sender sort and receiver merge boundaries. Done means distributed tests cover multiple senders, duplicate keys, nulls, and empty streams while cancellation, deadlines, backpressure, errors, and bounded memory remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.