apache / apache/pinot

[multistage][bug] block splitter estimation is way off

Open
#11,921 2 comments 0 reactions 0 assignees View on GitHub
bug multi-stage stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

when we send data over the mailboxes we are estimating the data size and cut the inbound messges into chunks. however

```
// Use estimated row size, this estimate is not accurate and is used to estimate numRowsPerChunk only.
int estimatedRowSizeInBytes = block.getDataSchema().getColumnNames().length * MEDIAN_COLUMN_SIZE_BYTES;
int numRowsPerChunk = maxBlockSize / estimatedRowSizeInBytes;
while (currentRow < totalNumRows) {
List chunk = allRows.subList(currentRow, Math.min(currentRow + numRowsPerChunk, allRows.size()));
```

this `estimatedRowSizeInBytes` can be WAY OFF when there's high-cardinality string/bytes column, and those columns are super large.

simple solution is to use the first row to estimate the size of the row when there's variable length columns found, but
- there's no easy way to tell cardinality
- it is expensive to compute a row size of `Object[]` which needs to loop through everything.

Contributor guide

Open the contributing guide

Research direction

Locate the multistage block splitter and the estimation using getDataSchema(), MEDIAN_COLUMN_SIZE_BYTES, and maxBlockSize. Reproduce the problem with a block containing a very large variable-length string or bytes column, then inspect existing splitter coverage and add a regression case. Done means chunk sizing is no longer badly underestimated while preserving the mailbox size limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.