apache / apache/pinot

[multistage] prevent OOM from broker/server QueryRunner that received too many data

Open
#12,309 1 comment 0 reactions 0 assignees View on GitHub
multi-stage stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Problem Statement
===
there are several execution path that could cause a legitimate OOM issue when executing a super large result set query
such as

```
select * FROM a JOIN b on a.boolCol = b.boolCol
```

this could result in multiplication result size compared to either table's row size. Say 1_000_000 rows on `a` join 1_000_000 on `b` on a 50-50 true/false split; this is under the default join max row limit but can produce a estimated result of 250B row!

Fix Idea
===

We need to safeguard against these types of potential OOM. this not only will cause the servers to crash but also incur very large GC overhead when these types of queries are running. specifically,
1. The broker is caching all data and returns back all at once. which is most likely to OOM
2. we don't have a streaming SortOperator so potentially a singleton server that does the final stage ORDER BY can also get OOM.

Related
====
#12304 addresses this partially by not allowing these types of non-limited queries to be accidentally run; but this one is to fix it more fundamentally in a safe-guard manner.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the broker/server QueryRunner paths that cache and return complete result sets, then examine the final-stage ORDER BY path and the non-streaming SortOperator. Define and test a safeguard for very large result sets so legitimate joins cannot cause OOM or excessive GC overhead, while accounting for the partial protection in #12304.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.