googleapis / googleapis/google-cloud-rust
feat(storage): GCS turn on upload checksum for Seek data sources by default
- Dominant language
- Rust
- Stars
- 955
- Forks
- 144
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 279
Description
**Is your feature request related to a problem? Please describe.**
Currently, when performing unbuffered, single-shot object uploads via the JSON (REST) API, the payload is streamed continuously in a single HTTP `PUT` request for maximum performance. However, because HTTP/1.1 requires all headers to be sent *before* the request body, we cannot append an on-the-fly computed CRC32C or MD5 checksum via the traditional `x-goog-hash` header. Achieving this currently requires passing over the payload twice (once to pre-compute the checksum and once to stream the data), creating a significant disk I/O performance penalty.
**Describe the solution you'd like**
To provide server-side validation against network data corruption without the double-read overhead, we should leverage the GCS JSON API's multipart trailing metadata feature.
**Implementation Constraints**
This trailing checksum strategy should be mutually exclusive with other kinds of checksum validation opt-in'ed by users.
b/521188176
Contributor guide
Assessment
This issue has not been assessed yet.