[Proposal] Native, vectorised, zero-copy execution path for Druid
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Motivation
Native, vectorised, zero-copy execution path for Druid via FFM and Arrow — staged from segment reader → filter/project → DataFusion (follow-up to #19039 item 1)
Druid today does the entire query hot path on the JVM. Each row passes through a ColumnSelector interface, materialises into Java objects, gets compared/aggregated by JIT-compiled byte code - means :
* No SIMD on filter / project / aggregate hot loops.
* GC pressure from per-block wrapper objects on cold-cache scans.
* 2 GiB ceiling on off-heap buffers (see #19025).
* Bottleneck on MSQ joins / aggregates
### Proposed changes
Phase 0 — Spike: native segment column reader via FFM
Phase 1 — Layer0 production-shape: native column reader for all common types
Phase 2 — L1: vectorised native filter + project over Arrow batches
Phase 3 — L2: DataFusion as candidate aggregate/join executor in MSQ stages
### Rationale
### Operational impact
Performance 2–3× speedup on
* speedup on cold-cache scan + filter for typical Druid datasources.
* speedup on MSQ fact-fact joins.
* speedup on external Parquet ingestion.
* lower p99 for high-cardinality GroupBy.
* faster execution under memory pressure (DataFusion graceful spill).
Operational
* 2 GiB ByteBuffer ceiling lifted on the native path (aligns with #19025).
* Reduced GC pressure → fewer p99 latency spikes.
### Test plan (optional)
### Future work (optional)
Contributor guide
Research direction
Start by reading follow-up issue #19039 item 1 and the 2 GiB buffer context in #19025. The proposal describes phases from a native segment reader through vectorised filtering and possible DataFusion execution, but names no files, entry points, tests, or acceptance criteria; those would need to be established before implementation can be considered done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100