Implement Non-contiguous Memory Buffer to Optimize Data Transfer
- Dominant language
- Rust
- Stars
- 12.5k
- Forks
- 1.3k
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 24
Description
## Feature Request: Implement Non-contiguous Memory Buffer to Optimize Data Transfer
### Crates
* `tonic`
* `tonic-build`
### Motivation
Many well-adopted `gRPC` use cases, such as [Arrow Flight](https://arrow.apache.org/docs/format/Flight.html), utilize `gRPC` to transfer large data chunks. In essence, Arrow Flight employs the `gRPC` protocol for network data transfer with the following protobuf definition:
```protobuf
message FlightData {
FlightDescriptor flight_descriptor = 1;
bytes data_header = 2;
bytes app_metadata = 3;
bytes data_body = 1000;
}
```
Here, `FlightDescriptor` is another protobuf message defined by Arrow Flight, which is not detailed here for brevity. The `data_body` field, which typically holds large payloads, contrasts with the other three fields that are relatively small. Arrow Flight is recognized for its high performance in transferring large data chunks, with specialized and optimized implementations in languages like C++ and JAVA.
In Rust, when using `tonic `for Arrow Flight, I observed a potential improvement area related to memory copying. The default codec in `tonic` utilizes `bytes::BytesMut`, essentially a contiguous memory chunk. Let's delve into the specifics:
#### Encoder
The default implementation directly copies the `data_body` into the buffer, leading to a `memcpy` syscall over a large memory segment and triggering memory reallocation.
#### Decoder
Upon polling a data chunk from the gRPC response body, the decoder directly transfers this data to the buffer, which again causes memory copying and reallocation.
### Proposal
#### Non-contiguous memory buffer
Rather than relying on `bytes::BytesMut` in the codec, we should introduce a specialized buffer backed by non-contiguous memory. The C++ gRPC solution has a [`SliceBuffer`](https://github.com/grpc/grpc/blob/6c11f4f181fef6b1a5a857b92dbeadbb7855a3b8/src/core/lib/slice/slice_buffer.h#L50) structure, comprising a set of slices (contiguous memory segments). We can take inspiration from this and create a Rust counterpart:
```Rust
pub struct SliceBuffer {
active: BytesMut,
len: usize,
slices: VecDeque,
}
```
Here, `SliceBuffer` portrays bytes as a concatenated sequence of all slices combined with the active. Additionally, we'd implement necessary traits, including `bytes::Buf` and `bytes::BufMut`, ensuring seamless read/write operations.
#### Encoder
The default protobuf encoder logic remains unchanged: bytes are read/written to the `active` field of the `SliceBuffer`. For specific scenarios like Arrow Flight, users would implement custom codecs. Using Arrow Flight as an example, the custom codec would directly append the `data_body` bytes of `FlightData` to the `slices` field of SliceBuffer, thus eliminating memory copying.
Furthermore, by implementing the [`chunks_vectored`](https://github.com/tokio-rs/bytes/blob/master/src/buf/buf_impl.rs#L181-L192) method for `SliceBuffer`, the underlying transport can directly send the `SliceBuffer`, capitalizing on vectored I/O and preventing `data_body` memory copying.
#### Decoder
When the decoder polls a data chunk from the gRPC response body, it appends it directly to the `slices` of `SliceBuffer`. Avoiding memory copying in situations with large data chunks can significantly boost performance.
#### Drawback
For use cases with smaller data volumes, memory copying isn't the primary performance concern. However, the proposed solution might introduce slight overhead in such scenarios compared to the direct usage of `bytes::BytesMut`. Even so, this minor overhead from `SliceBuffer` would likely not emerge as a new bottleneck.
### Alternatives
One alternative is to place the `SliceBuffer` behind a feature flag, retaining `tonic`'s default behavior. Based on the proposal, the encoder's default behavior remains unchanged unless a custom codec is specified, but the default decoder changes to avoid memory copying by inserting the slice directly into the `SliceBuffer`.
My benchmarks show decoder performance improvements in many scenarios, especially those mirroring real-world use cases. Adding a feature guard might confuse users and detract from codebase readability, so I recommend against it.
Contributor guide
Assessment
This issue has not been assessed yet.