apache / apache/opendal

CompleteReader rejects a bounded range that extends past EOF

Open
#7,938 7 comments 1 reaction 0 assignees View on GitHub
bug releases-note/fix services/fs
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

Somewhere between opendal 0.45 and 0.57 (EDIT: I "bisected" this to have happened [between 0.45 and 0.47](https://github.com/apache/opendal/compare/v0.45.0...v0.47.3)!), the semantics for a short read changed: in 0.57, a bounded range read whose end extends past the end of the object fails with

```
Unexpected (permanent) => reader got too little data { expect: , actual: }
```

instead of returning the available bytes (a short read).

This is a behavior change from earlier releases, and it breaks the common "over-request a header window without first knowing the object size" pattern.

Standard HTTP Range semantics already clamp this: `Range: bytes=0-16383` against a 1088-byte object returns `206` with `Content-Range: bytes 0-1087/1088`. `CompleteLayer` ignores what was actually served and validates against the requested length instead, so it also fires for the `fs` service where there is no HTTP layer at all.

## Minimal reproduction (`services-fs`)

```rust
use opendal::{services::Fs, Operator};

#[tokio::main]
async fn main() {
let dir = std::env::temp_dir().join("opendal-range-repro");
std::fs::create_dir_all(&dir).unwrap();
let op = Operator::new(Fs::default().root(dir.to_str().unwrap()))
.unwrap()
.finish();

// Object is 100 bytes.
op.write("small", vec![b'x'; 100]).await.unwrap();

// Ask for the first 16 KiB. The object is shorter than that.
let res = op.read_with("small").range(0..16_384).await;

match res {
Ok(buf) => println!("ok: {} bytes", buf.len()), // expected: 100
Err(e) => println!("ERR: {e}"), // actual (0.57): reader got too little data
}
}
```

Output on 0.57 (verified):

```
ERR: Unexpected (permanent) at , context: { expect: 16384, actual: 100 } => reader got too little data
```

## Expected

A range whose end runs past EOF returns the available bytes (here, 100), as it did before and as HTTP Range semantics imply. Reading the first N bytes of an object of unknown length should preferably not require a preceding `stat`, or at least there should be an option to allow for it?

### AI Disclosure

This was partially triaged, and the verifier written, by Claude Opus 4.8.

Contributor guide

Open the contributing guide

Research direction

Locate the CompleteReader/CompleteLayer implementation and start by tracing how a bounded range is validated after the fs service returns a short read. Use the services-fs reproduction with a 100-byte object and a 0..16,384 range, then add or update coverage for the expected 100-byte result. Done means the read succeeds with the available bytes instead of returning the permanent short-data error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.