googleapis / googleapis/google-cloud-rust

feat(storage): GCS turn on upload checksum for Seek data sources by default

Open
#6,461 1 comment 0 reactions 1 assignee Claimed by @xlai20 View on GitHub
api: storage priority: p3 type: feature request
Dominant language
Rust
Stars
955
Forks
144
Avg merge
1d 5h
Merged PRs (30d)
279

Description

**Is your feature request related to a problem? Please describe.**
Currently, when performing unbuffered, single-shot object uploads via the JSON (REST) API, the payload is streamed continuously in a single HTTP `PUT` request for maximum performance. However, because HTTP/1.1 requires all headers to be sent *before* the request body, we cannot append an on-the-fly computed CRC32C or MD5 checksum via the traditional `x-goog-hash` header. Achieving this currently requires passing over the payload twice (once to pre-compute the checksum and once to stream the data), creating a significant disk I/O performance penalty.

**Describe the solution you'd like**
To provide server-side validation against network data corruption without the double-read overhead, we should leverage the GCS JSON API's multipart trailing metadata feature.

**Implementation Constraints**
This trailing checksum strategy should be mutually exclusive with other kinds of checksum validation opt-in'ed by users.

b/521188176

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.