apache / apache/druid

MSQ: introduce ArrowRowsAndColumns and adopt it inside GroupByPostShuffleFrameProcessor

Open
#19,499 0 comments 0 reactions 0 assignees View on GitHub
Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description
Introduce Apache Arrow as a first-class in-memory representation inside Druid's MSQ engine by adding ArrowRowsAndColumns .

The #19456 discussion identified that swapping Druid's row-at-a-time JVM hot path requires touching multiple load-bearing abstractions (Frame, FrameProcessor, channels, leaf segment readers, shuffle, planner).

As @gianm , @jtuglu1 mentioned - The first move is the operator with the smallest blast radius (one non-leaf single-input FrameProcessor) backed by a reusable abstraction (ArrowRowsAndColumns) — so every new implementations phases reuses it.

### Motivation

introduce ArrowRowsAndColumns and adopt it inside GroupByPostShuffleFrameProcessor — first concrete step of the #19456 modernisation program.

Related

• #19456 — Native, vectorised, zero-copy execution path (this is the first concrete step)
• #13458 — RowsAndColumns introduction (the abstraction this extends)
• #18909 — WireTransferable (the seam that Phase C will use)

Contributor guide

Open the contributing guide

Research direction

Start by reading the #19456 discussion, then review the RowsAndColumns abstraction from #13458 and the WireTransferable seam in #18909. The target is GroupByPostShuffleFrameProcessor: done means introducing ArrowRowsAndColumns and adopting it there as the first concrete MSQ modernization step.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.