[multistage][bug] block splitter estimation is way off
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
when we send data over the mailboxes we are estimating the data size and cut the inbound messges into chunks. however
```
// Use estimated row size, this estimate is not accurate and is used to estimate numRowsPerChunk only.
int estimatedRowSizeInBytes = block.getDataSchema().getColumnNames().length * MEDIAN_COLUMN_SIZE_BYTES;
int numRowsPerChunk = maxBlockSize / estimatedRowSizeInBytes;
while (currentRow < totalNumRows) {
List chunk = allRows.subList(currentRow, Math.min(currentRow + numRowsPerChunk, allRows.size()));
```
this `estimatedRowSizeInBytes` can be WAY OFF when there's high-cardinality string/bytes column, and those columns are super large.
simple solution is to use the first row to estimate the size of the row when there's variable length columns found, but
- there's no easy way to tell cardinality
- it is expensive to compute a row size of `Object[]` which needs to loop through everything.
Contributor guide
Research direction
Locate the multistage block splitter and the estimation using getDataSchema(), MEDIAN_COLUMN_SIZE_BYTES, and maxBlockSize. Reproduce the problem with a block containing a very large variable-length string or bytes column, then inspect existing splitter coverage and add a regression case. Done means chunk sizing is no longer badly underestimated while preserving the mailbox size limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100