apache / apache/arrow-java

Use default chunk size as 4 MB than than 16Mb in arrow's rolling policy

Open
#1,040 0 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
94
Forks
152
Avg merge
3d 16h
Merged PRs (30d)
11

Description

### Describe the enhancement requested

Apache Arrow currently calculates chunk sizes based on the assumption that Netty's PooledByteBufAllocatorL uses a 16MB page size. If size is blow 16MB, will use nextPowerOfTwo of requestSize.

```java
public long getRoundedSize(long requestSize) {
return requestSize < chunkSize ? CommonUtil.nextPowerOfTwo(requestSize) : requestSize;
}
```
However, Netty's default maximum buffer size has been reduced from 16MB to 4MB in recent versions (see https://github.com/netty/netty/pull/12108). This mismatch leads to memory inefficiency in scenarios where chunk sizes fall between 4MB and 16MB.

Contributor guide

Open the contributing guide

Research direction

Start by locating the rolling policy implementation containing getRoundedSize and the chunkSize default, then read how CommonUtil.nextPowerOfTwo is used. Compare the policy with Netty's 4 MB default described in the issue. Done means the default uses 4 MB and the existing behavior for request sizes is covered by the relevant tests, if present.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.