Use default chunk size as 4 MB than than 16Mb in arrow's rolling policy
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the enhancement requested
Apache Arrow currently calculates chunk sizes based on the assumption that Netty's PooledByteBufAllocatorL uses a 16MB page size. If size is blow 16MB, will use nextPowerOfTwo of requestSize.
```java
public long getRoundedSize(long requestSize) {
return requestSize < chunkSize ? CommonUtil.nextPowerOfTwo(requestSize) : requestSize;
}
```
However, Netty's default maximum buffer size has been reduced from 16MB to 4MB in recent versions (see https://github.com/netty/netty/pull/12108). This mismatch leads to memory inefficiency in scenarios where chunk sizes fall between 4MB and 16MB.
Contributor guide
Research direction
Start by locating the rolling policy implementation containing getRoundedSize and the chunkSize default, then read how CommonUtil.nextPowerOfTwo is used. Compare the policy with Netty's 4 MB default described in the issue. Done means the default uses 4 MB and the existing behavior for request sizes is covered by the relevant tests, if present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100