grpc / grpc/grpc-rust

Implement Non-contiguous Memory Buffer to Optimize Data Transfer

Open
#1,558 4 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
12.5k
Forks
1.3k
Avg merge
4d 7h
Merged PRs (30d)
24

Description

## Feature Request: Implement Non-contiguous Memory Buffer to Optimize Data Transfer

### Crates

* `tonic`
* `tonic-build`

### Motivation

Many well-adopted `gRPC` use cases, such as [Arrow Flight](https://arrow.apache.org/docs/format/Flight.html), utilize `gRPC` to transfer large data chunks. In essence, Arrow Flight employs the `gRPC` protocol for network data transfer with the following protobuf definition:

```protobuf
message FlightData {
FlightDescriptor flight_descriptor = 1;
bytes data_header = 2;
bytes app_metadata = 3;
bytes data_body = 1000;
}
```

Here, `FlightDescriptor` is another protobuf message defined by Arrow Flight, which is not detailed here for brevity. The `data_body` field, which typically holds large payloads, contrasts with the other three fields that are relatively small. Arrow Flight is recognized for its high performance in transferring large data chunks, with specialized and optimized implementations in languages like C++ and JAVA.

In Rust, when using `tonic `for Arrow Flight, I observed a potential improvement area related to memory copying. The default codec in `tonic` utilizes `bytes::BytesMut`, essentially a contiguous memory chunk. Let's delve into the specifics:

#### Encoder

The default implementation directly copies the `data_body` into the buffer, leading to a `memcpy` syscall over a large memory segment and triggering memory reallocation.

#### Decoder

Upon polling a data chunk from the gRPC response body, the decoder directly transfers this data to the buffer, which again causes memory copying and reallocation.

### Proposal

#### Non-contiguous memory buffer

Rather than relying on `bytes::BytesMut` in the codec, we should introduce a specialized buffer backed by non-contiguous memory. The C++ gRPC solution has a [`SliceBuffer`](https://github.com/grpc/grpc/blob/6c11f4f181fef6b1a5a857b92dbeadbb7855a3b8/src/core/lib/slice/slice_buffer.h#L50) structure, comprising a set of slices (contiguous memory segments). We can take inspiration from this and create a Rust counterpart:

```Rust
pub struct SliceBuffer {
active: BytesMut,
len: usize,
slices: VecDeque,
}
```

Here, `SliceBuffer` portrays bytes as a concatenated sequence of all slices combined with the active. Additionally, we'd implement necessary traits, including `bytes::Buf` and `bytes::BufMut`, ensuring seamless read/write operations.

#### Encoder

The default protobuf encoder logic remains unchanged: bytes are read/written to the `active` field of the `SliceBuffer`. For specific scenarios like Arrow Flight, users would implement custom codecs. Using Arrow Flight as an example, the custom codec would directly append the `data_body` bytes of `FlightData` to the `slices` field of SliceBuffer, thus eliminating memory copying.

Furthermore, by implementing the [`chunks_vectored`](https://github.com/tokio-rs/bytes/blob/master/src/buf/buf_impl.rs#L181-L192) method for `SliceBuffer`, the underlying transport can directly send the `SliceBuffer`, capitalizing on vectored I/O and preventing `data_body` memory copying.

#### Decoder

When the decoder polls a data chunk from the gRPC response body, it appends it directly to the `slices` of `SliceBuffer`. Avoiding memory copying in situations with large data chunks can significantly boost performance.

#### Drawback

For use cases with smaller data volumes, memory copying isn't the primary performance concern. However, the proposed solution might introduce slight overhead in such scenarios compared to the direct usage of `bytes::BytesMut`. Even so, this minor overhead from `SliceBuffer` would likely not emerge as a new bottleneck.

### Alternatives

One alternative is to place the `SliceBuffer` behind a feature flag, retaining `tonic`'s default behavior. Based on the proposal, the encoder's default behavior remains unchanged unless a custom codec is specified, but the default decoder changes to avoid memory copying by inserting the slice directly into the `SliceBuffer`.

My benchmarks show decoder performance improvements in many scenarios, especially those mirroring real-world use cases. Adding a feature guard might confuse users and detract from codebase readability, so I recommend against it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.