grpc / grpc/grpc-rust

metadata: Changes to support gRPC Rust requirements

Open
#2,487 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
12.5k
Forks
1.3k
Avg merge
4d 7h
Merged PRs (30d)
24

Description

## Feature Request

gRPC Rust requires a Metadata API that is consistent with the [gRPC over HTTP/2 protocol](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md) and capable of supporting [true binary metadata (gRFC G1)](https://github.com/grpc/proposal/blob/master/G1-true-binary-metadata.md) in the future.

While evaluating the feasibility of reusing tonic's [MetadataMap](https://github.com/hyperium/tonic/blob/a58c291181fe8866a1fb57a20794a440ede20c83/tonic/src/metadata/map.rs#L37-L39), I identified two major areas requiring modification:
1. Lazy encoding of binary values.
2. Stricter validation of header keys and values.

### Motivation

In gRPC, binary metadata (keys ending with the `-bin` suffix) must be base64 encoded because the HTTP/2 specification discourages specific special characters in header values.

[gRFC G1](https://github.com/grpc/proposal/blob/master/G1-true-binary-metadata.md) describes an extension to the gRPC over HTTP/2 protocol where peers can use the HTTP/2 `Settings` frame to indicate support for raw (non-base64 encoded) metadata. This eliminates the CPU overhead of base64 encoding binary data. To implement this, the sender must inspect the peer's `Settings` frame to decide whether or not to encode the data.

Currently, `MetadataMap` encodes data immediately upon insertion into the map, incurring the CPU cost regardless of the peer's capabilities.

### Proposal

#### Lazily encode binary data

To support gRFC G1 in the future, we must modify the internal structure of `MetadataMap` to store unencoded header values. This could remain an `http::HeaderMap`, where `UnencodedHeaderValue` wraps the raw bytes.

The [MetadataMap::into_headers](https://github.com/hyperium/tonic/blob/a58c291181fe8866a1fb57a20794a440ede20c83/tonic/src/metadata/map.rs#L261-L263) and [MetadataMap::from_headers](https://github.com/hyperium/tonic/blob/a58c291181fe8866a1fb57a20794a440ede20c83/tonic/src/metadata/map.rs#L244-L246) methods would be updated to convert the internal `http::HeaderMap` to/from an `http::HeaderMap`.

Because the encoded data would no longer be stored directly in the `MetadataMap`, APIs that provide a reference to encoded data must be removed. While they could theoretically be retained for `Ascii` values, keeping them for binary values would require returning unencoded data, which constitutes a larger behaviour change.

**APIs to remove:**
* [MetadataValue::as_encoded_bytes](https://docs.rs/tonic/latest/tonic/metadata/struct.MetadataValue.html#method.as_encoded_bytes)
* [impl AsRef<[u8]> for MetadataValue](https://docs.rs/tonic/latest/tonic/metadata/struct.MetadataValue.html#impl-AsRef%3C%5Bu8%5D%3E-for-MetadataValue%3CVE%3E)

**Trait implementations to remove:**
The following implementations expose the internal `http::HeaderMap`. These must be removed because the map will now hold unencoded values requiring conversion:
* [impl AsMut for MetadataMap](https://docs.rs/tonic/latest/tonic/metadata/struct.MetadataMap.html#impl-AsMut%3CHeaderMap%3E-for-MetadataMap)
* [impl AsRef for MetadataMap](https://docs.rs/tonic/latest/tonic/metadata/struct.MetadataMap.html#impl-AsRef%3CHeaderMap%3E-for-MetadataMap)

#### Stricter header validation

To ensure compliance with the [gRPC over HTTP/2 protocol](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md), the following changes regarding header validation are required:

1. **ASCII Values:** Disallow extended ASCII characters (128-255).
2. **ASCII Values:** Strip leading and trailing whitespace.
3. **Key Validation:** Header keys must only contain characters from the set `0-9 a-z _ - .`.
4. **Import Logic:** In `MetadataMap::from_headers`, headers containing extended ASCII characters (128-255) or keys with invalid characters will be dropped/ignored.

### Alternatives

I could implement a simplified `MetadataMap` for gRPC Rust that exposes an API similar to `HashMap>` while hiding the internal data structure. For comparison, gRPC C++ and Go do not provide typed APIs for ASCII and binary metadata, instead operating directly on strings.

The following shows the proposed changes: https://github.com/arjan-bal/tonic/pull/8

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.