[enhancement ] Reusing the lengths array in ArrowFieldWrites$ArrayWriter's doWrite method
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
ArrowFieldWriters$ArrayWriter#doWrite

### Solution
### **Implementation Steps**
1. **Reusable Buffer Declaration**
Define a reusable buffer field in `ArrayWriter` class:
```java
private int[] reusedLengthsBuffer; // Reusable array buffer for length storage
```
2. **Lazy Initialization & Dynamic Expansion**
In `doWrite` method, initialize or expand buffer based on runtime `lenSize` requirements:
```java
if (reusedLengthsBuffer == null || reusedLengthsBuffer.length < lenSize) {
int newCapacity = lenSize;
// Apply 1.5x growth factor if reallocating from existing buffer
if (reusedLengthsBuffer != null) {
newCapacity = Math.max(lenSize, (int)(reusedLengthsBuffer.length * 1.5));
}
reusedLengthsBuffer = new int[newCapacity]; // Allocate or expand buffer
}
// Reset buffer segment before reuse
Arrays.fill(reusedLengthsBuffer, 0, lenSize, 0);
```
Let me know if you have any questions. thanks
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate ArrowFieldWriters$ArrayWriter and read its doWrite method to understand how the lengths array is currently allocated and consumed. Implement the reusable buffer and expansion behavior described in the issue, then verify that array writing retains its existing results and passes the repository's relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100