new feature: caller-owned buffer reads (fill a destination buffer directly)
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
# Feature Description
Add an optional read path that fills a **caller-owned destination buffer** directly, so that capable services and transports can write bytes into memory the caller already owns without OpenDAL first materializing an intermediate
owned `Buffer`/`Bytes`.
Today OpenDAL's reader pipeline always yields owned `Buffer`/`Bytes` chunks. The closest public API, `Reader::read_into(&mut impl BufMut, range)`, still consumes those owned chunks from the reader stream and copies them into the
caller's `BufMut`:
```rust
pub async fn read_into(
&self,
buf: &mut impl BufMut,
range: impl Into,
) -> Result {
let mut stream = self.clone().into_stream(range).await?;
let mut read = 0;
loop {
let Some(bs) = stream.try_next().await? else {
return Ok(read);
};
read += bs.len();
buf.put(bs); // copy into caller memory
}
}
```
With `&mut [u8]` as the `BufMut`, `bytes::BufMut::put_slice` performs a `copy_from_slice` into the caller's allocation. So the effective path is:
```text
backend/transport -> OpenDAL Buffer/Bytes -> caller-owned buffer
one memcpy
```
The request is for an API where the caller supplies the destination and, on
capable backends, the backend writes into it directly:
```text
backend/transport -> caller-owned destination (no intermediate data buffer)
```
# Problem and Solution
## Problem
Many callers do not own the shape of their destination memory — it is dictated by a foreign API, a hardware constraint, or a preallocated pool — and they require OpenDAL to *fill that specific memory*. In these cases OpenDAL's `Buffer`-first model forces an extra copy that has no semantic purpose.
Concrete cases where the caller owns the destination and cannot adopt an OpenDAL `Buffer`:
- **FFI / foreign filesystem bridges.** C/C++ `pread`-style contracts (`read(void *buffer, size_t nr_bytes, off_t offset)`) hand OpenDAL a raw destination pointer by value and require it to be filled. The buffer is owned and freed by the foreign runtime; OpenDAL cannot return or substitute its own `Buffer`, so today every such bridge pays a memcpy per read.
- **Columnar / Parquet readers.** Arrow-style readers assemble column chunks into preallocated, correctly-aligned buffers (often part of a larger arena or a `MutableBuffer`). They want positioned range reads landed straight into those buffers rather than into a transient `Buffer` that is immediately copied and dropped.
- **Buffer pools and arenas.** Query engines and network servers read into slabs from a fixed-capacity pool to bound allocation and improve cache locality. A `Buffer`-returning read defeats the pool because the data lands in OpenDAL-owned memory first.
- **Special-purpose memory.** Destinations such as `mmap` regions, GPU pinned host memory, `O_DIRECT`-aligned buffers, or `io_uring`-registered buffers must be written in place. They cannot be replaced by an OpenDAL allocation.
The extra copy is most visible exactly where network latency does *not* hide memory bandwidth: local `fs`, fast attached storage, the in-memory `memory` service, and cache hits (e.g. a warm cache layer). For large scans, many concurrent positioned reads, or high-throughput links, the redundant copy shows up as CPU cycles, memory bandwidth, and allocator pressure.
## Proposed solution
Introduce a caller-owned buffer read at two levels, gated by capability with a safe fallback:
1. **Public high-level method** on `Reader`, e.g.
```rust
/// Read exactly `dst.len()` bytes for `range` into caller-owned memory.
pub async fn read_exact_into(
&self,
dst: &mut [u8],
range: Range,
) -> Result<()>;
```
with a fixed-capacity, `pread`-like contract that matches positioned reads.
2. **Optional raw capability** on the positioned-read path so capable services fill the destination directly, for example an additive method alongside the existing `PositionRead::read_at`:
```rust
async fn read_at_into(
handle: &Self::Handle,
offset: u64,
dst: &mut [u8],
) -> Result;
```
Services that can write into a foreign destination (starting with `fs` via platform positioned reads, and `memory` via `copy_from_slice`) implement it. Everything else falls back to the existing `read_at` + one copy, so no service
is required to change and behavior is preserved.
The existing `read`/`read_into`/`fetch`/stream APIs and their zero-copy-within-OpenDAL semantics stay exactly as they are; this is purely additive.
A pragmatic first increment: land the raw capability + fs implementation + generic fallback + the `Reader` method, benchmark it against `read_into`, then consider extending to transports and other services. A full design (capability
advertisement, layer composition — retry/timeout/cancellation with borrowed memory, short-read/EOF semantics, `MaybeUninit` vs initialized slices) is worth an RFC; i'll be happy to draft that if this issue makes sense.
# Additional Context
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start with Reader::read_into and the positioned-read path around PositionRead::read_at, then inspect the fs and memory service implementations. Define the raw capability, generic fallback, and new Reader method while preserving existing APIs and semantics. Add benchmarks comparing the new path with read_into; completion also requires resolving short-read, cancellation, layering, and capability-design questions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100