apache / apache/opendal

new feature: expose seekable and positioned streaming reads in Java

Open
#8,253 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Support Java consumers that need seek, positioned reads, or tail reads without materializing an entire object or implementing their own S3 requests.

### Problem and Solution

A concrete caller is Apache Iceberg S3InputStream: it implements seek, readFully(position,...), and readTail(...). It reuses a current response for small forward seeks and otherwise opens a stream at the desired offset. Its range helpers return InputStreams rather than full-object arrays.

OpenDAL Java exposes createInputStream and whole-range read, but OperatorInputStream extends InputStream without a seek/positioned-read interface. Whole-range byte arrays alone are not equivalent to a reusable seekable stream. The Rust core already has Reader and seek-capable adapters; reuse them where their contract fits.

Acceptance criteria:
- Define cursor behavior for seek and independent positioned reads, including whether positioned reads change the sequential cursor.
- Support bounded memory, EOF/short reads, close, and failure propagation.
- Preserve relevant version/conditional-read guarantees across multiple requests; report gaps instead of silently mixing object versions.
- Avoid requiring an unconditional metadata request when an existing sequential read does not need length.
- Validate through an actual format reader with a recorded query/read trace. Do not invent uniformly spaced ranges and call them an Iceberg workload.

This is an additive capability/design request related to #6226, #7909, and #8251. It does not require taking a dependency on Iceberg or promise query acceleration before measurement.

### Additional Context

[Source reference](https://github.com/apache/iceberg/blob/af86b8399ca794bfe482eae85ba7e79ab0c49adc/aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java#L111). OpenDAL API inventory was checked at `b6cf44f7b8a1523409e0e998e478c996ac970f03`.

Contributor guide

Open the contributing guide

Research direction

Start with Java's OperatorInputStream and the Rust core's Reader and seek-capable adapters, then compare their contracts with Apache Iceberg's S3InputStream reference. Validate the design through an actual format reader and recorded query/read trace. Done means bounded-memory seek and positioned reads with defined cursor, EOF, close, failure, and version-consistency behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.