apache / apache/paimon

[enhancement ] Reusing the lengths array in ArrowFieldWrites$ArrayWriter's doWrite method

Open
#5,401 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

ArrowFieldWriters$ArrayWriter#doWrite

![Image](https://github.com/user-attachments/assets/a82c8089-f37d-4ae4-b859-90dca2babee6)

### Solution

### **Implementation Steps**
1. **Reusable Buffer Declaration**
Define a reusable buffer field in `ArrayWriter` class:
```java
private int[] reusedLengthsBuffer; // Reusable array buffer for length storage
```

2. **Lazy Initialization & Dynamic Expansion**
In `doWrite` method, initialize or expand buffer based on runtime `lenSize` requirements:
```java
if (reusedLengthsBuffer == null || reusedLengthsBuffer.length < lenSize) {
int newCapacity = lenSize;
// Apply 1.5x growth factor if reallocating from existing buffer
if (reusedLengthsBuffer != null) {
newCapacity = Math.max(lenSize, (int)(reusedLengthsBuffer.length * 1.5));
}
reusedLengthsBuffer = new int[newCapacity]; // Allocate or expand buffer
}

// Reset buffer segment before reuse
Arrays.fill(reusedLengthsBuffer, 0, lenSize, 0);
```

Let me know if you have any questions. thanks

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate ArrowFieldWriters$ArrayWriter and read its doWrite method to understand how the lengths array is currently allocated and consumed. Implement the reusable buffer and expansion behavior described in the issue, then verify that array writing retains its existing results and passes the repository's relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.