apache / apache/opendal

new feature: avoid synchronous lookahead after Java InputStream fills the destination

Open
#8,245 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Return a completed `OperatorInputStream.read(byte[], off, len)` without fetching the next native chunk first.

### Problem and Solution

After the copy loop consumes the requested length, the method calls `readNextBytes` again if the current chunk is exhausted. The destination is already full, but the caller can still wait for another storage read. This adds avoidable latency when the next chunk is slow.

The next native read could be deferred until the next call actually needs data. Preserve positive byte counts, EOF/error ordering, zero-length reads, and close behavior.

Acceptance criteria:
- Add a deterministic test with a ready first chunk and a blocked second chunk: a destination exactly matching the first chunk must be returned before releasing the second chunk.
- Verify subsequent bytes and EOF, partial reads, zero-length reads, and errors.
- Measure first-consumer-byte latency separately from aggregate throughput.

### Additional Context

[Bulk read and trailing readNextBytes](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/java/src/main/java/org/apache/opendal/OperatorInputStream.java#L61)

This concerns generic Java stream scheduling, not a service-specific S3 failure. It does not require buffering a complete object or changing core transport concurrency.

Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.

Contributor guide

Open the contributing guide

Research direction

Start with OperatorInputStream.read(byte[], off, len) and readNextBytes in bindings/java/src/main/java/org/apache/opendal/OperatorInputStream.java at the referenced commit. Add a deterministic test using a ready first chunk and blocked second chunk, then verify subsequent bytes, EOF, partial and zero-length reads, errors, close behavior, and separate first-byte latency from aggregate throughput.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.