CompleteReader rejects a bounded range that extends past EOF
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
Somewhere between opendal 0.45 and 0.57 (EDIT: I "bisected" this to have happened [between 0.45 and 0.47](https://github.com/apache/opendal/compare/v0.45.0...v0.47.3)!), the semantics for a short read changed: in 0.57, a bounded range read whose end extends past the end of the object fails with
```
Unexpected (permanent) => reader got too little data { expect: , actual: }
```
instead of returning the available bytes (a short read).
This is a behavior change from earlier releases, and it breaks the common "over-request a header window without first knowing the object size" pattern.
Standard HTTP Range semantics already clamp this: `Range: bytes=0-16383` against a 1088-byte object returns `206` with `Content-Range: bytes 0-1087/1088`. `CompleteLayer` ignores what was actually served and validates against the requested length instead, so it also fires for the `fs` service where there is no HTTP layer at all.
## Minimal reproduction (`services-fs`)
```rust
use opendal::{services::Fs, Operator};
#[tokio::main]
async fn main() {
let dir = std::env::temp_dir().join("opendal-range-repro");
std::fs::create_dir_all(&dir).unwrap();
let op = Operator::new(Fs::default().root(dir.to_str().unwrap()))
.unwrap()
.finish();
// Object is 100 bytes.
op.write("small", vec![b'x'; 100]).await.unwrap();
// Ask for the first 16 KiB. The object is shorter than that.
let res = op.read_with("small").range(0..16_384).await;
match res {
Ok(buf) => println!("ok: {} bytes", buf.len()), // expected: 100
Err(e) => println!("ERR: {e}"), // actual (0.57): reader got too little data
}
}
```
Output on 0.57 (verified):
```
ERR: Unexpected (permanent) at , context: { expect: 16384, actual: 100 } => reader got too little data
```
## Expected
A range whose end runs past EOF returns the available bytes (here, 100), as it did before and as HTTP Range semantics imply. Reading the first N bytes of an object of unknown length should preferably not require a preceding `stat`, or at least there should be an option to allow for it?
### AI Disclosure
This was partially triaged, and the verifier written, by Claude Opus 4.8.
Contributor guide
Research direction
Locate the CompleteReader/CompleteLayer implementation and start by tracing how a bounded range is validated after the fs service returns a short read. Use the services-fs reproduction with a 100-byte object and a 0..16,384 range, then add or update coverage for the expected 100-byte result. Done means the read succeeds with the available bytes instead of returning the permanent short-data error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100