apache / apache/druid

[Proposal] Native, vectorised, zero-copy execution path for Druid

Open
#19,456 11 comments 3 reactions 0 assignees View on GitHub
Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

Native, vectorised, zero-copy execution path for Druid via FFM and Arrow — staged from segment reader → filter/project → DataFusion (follow-up to #19039 item 1)

Druid today does the entire query hot path on the JVM. Each row passes through a ColumnSelector interface, materialises into Java objects, gets compared/aggregated by JIT-compiled byte code - means :

* No SIMD on filter / project / aggregate hot loops.
* GC pressure from per-block wrapper objects on cold-cache scans.
* 2 GiB ceiling on off-heap buffers (see #19025).
* Bottleneck on MSQ joins / aggregates

### Proposed changes

Phase 0 — Spike: native segment column reader via FFM
Phase 1 — Layer0 production-shape: native column reader for all common types
Phase 2 — L1: vectorised native filter + project over Arrow batches
Phase 3 — L2: DataFusion as candidate aggregate/join executor in MSQ stages

### Rationale

### Operational impact

Performance 2–3× speedup on

* speedup on cold-cache scan + filter for typical Druid datasources.
* speedup on MSQ fact-fact joins.
* speedup on external Parquet ingestion.
* lower p99 for high-cardinality GroupBy.
* faster execution under memory pressure (DataFusion graceful spill).

Operational

* 2 GiB ByteBuffer ceiling lifted on the native path (aligns with #19025).
* Reduced GC pressure → fewer p99 latency spikes.

### Test plan (optional)

### Future work (optional)

Contributor guide

Open the contributing guide

Research direction

Start by reading follow-up issue #19039 item 1 and the 2 GiB buffer context in #19025. The proposal describes phases from a native segment reader through vectorised filtering and possible DataFusion execution, but names no files, entry points, tests, or acceptance criteria; those would need to be established before implementation can be considered done.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.