pingcap / pingcap/tidb

objstore: preserve storage class and ACL for concurrent GCS multipart uploads

Open
#70,007 6 comments 0 reactions 1 assignee Claimed by @joechenrh View on GitHub
severity/minor
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Summary
The GCS concurrent multipart-upload implementation does not preserve configured object metadata. When `pkg/objstore.GCSStorage.Create` is called with `WriterOption.Concurrency > 1`, it delegates to `NewGCSWriter`; unlike the serial GCS path, that writer does not receive or apply `StorageClass` and `PredefinedACL`.

## Rationale
This causes concurrent GCS uploads—such as Dumpling CSV/SQL uploads—to silently use default storage class and ACL settings. The S3 multipart implementation preserves equivalent options, so GCS behavior is inconsistent.

## Affected areas
- `pkg/objstore/gcs.go`
- `pkg/objstore/gcs_extra.go`
- GCS XML multipart-upload initiation request construction and request signing

## Required changes
1. Thread `WriterOption.StorageClass` and `WriterOption.PredefinedACL` from `GCSStorage.Create` into `NewGCSWriter`.
2. Add the corresponding `x-goog-storage-class` and `x-goog-acl` headers to the XML multipart initiate request when configured.
3. Ensure these headers are included in the signed request.
4. Add focused tests covering configured and unset metadata values for concurrent multipart initiation.

## Acceptance criteria
- Concurrent GCS multipart uploads preserve configured storage class and predefined ACL.
- The initiation request signs and sends the applicable GCS metadata headers.
- Unset options retain existing behavior.
- Focused regression tests cover the multipart-initiate path.

## Backlinks
- Triggering PR: https://github.com/pingcap/tidb/pull/69187
- Requesting comment: https://github.com/pingcap/tidb/pull/69187#issuecomment-5065379484
- Requested by: @joechenrh

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.