apache / apache/opendal

new feature: optimize bulk writes in Java OperatorOutputStream

Open
#8,244 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Make ordinary `OutputStream.write(byte[], off, len)` efficient for OpenDAL file uploads and stream copying.

### Problem and Solution

The implementation overrides only `write(int)`, so inherited bulk writes process the input one byte at a time. Its default staging buffer is 16 KiB, and every full buffer enters `writeBytes` through JNI. Large application writes therefore still pay per-byte Java dispatch and many native calls.

A bulk override could copy partial inputs with array operations, coalesce small writes, and forward suitable large arrays in larger native batches. The core writer should continue to own multipart formation and backpressure.

Acceptance criteria:
- Preserve offsets, lengths, mixed scalar/bulk writes, flush, close, and error behavior.
- Keep caller-array mutation safe after write returns, including pending uploads and retries.
- Verify partial-buffer boundaries and exact uploaded contents.
- Measure normal file-to-OutputStream copying against the unchanged implementation; report CPU, transfer latency, and memory separately.

### Additional Context

[Current buffer and write implementation](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/java/src/main/java/org/apache/opendal/OperatorOutputStream.java#L39)

Bulk dispatch and JNI batching are separate changes: a bulk override alone need not reduce native-call count. Increasing a buffer size is not evidence of a proportional throughput gain.

Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.

Contributor guide

Open the contributing guide

Research direction

Start with bindings/java/src/main/java/org/apache/opendal/OperatorOutputStream.java at the pinned buffer and write implementation, then trace the existing writeBytes JNI path and core writer behavior. Check how scalar and bulk writes interact with buffering, pending uploads, retries, flush, and close. Done means preserved contents and error semantics, safe caller-array handling, and reported CPU, latency, and memory measurements against the unchanged implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.