`google-cloud-storage`: background `BucketMetadataCache` request shares the upload transport and correlates with intermittent `SSLEOFError`

# `google-cloud-storage`: background `BucketMetadataCache` request shares the upload transport and correlates with intermittent `SSLEOFError`

## Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

This concerns client-side behavior introduced by `google-cloud-storage` 3.11.0, not the semantics of the Cloud Storage API.

## Summary of the issue

### Context

We run a Python service on Google Cloud Run that performs multipart object uploads to Google Cloud Storage. Upload calls are synchronous and execute in worker threads. We have tried both a client scoped to each worker thread and a new `storage.Client` for each upload.

After upgrading from `google-cloud-storage` 3.10.1 to 3.13.0, intermittent uploads began exhausting the normal 120-second retry deadline with this underlying error:

```text
google.api_core.exceptions.RetryError: Timeout of 120.0s exceeded, last exception:
HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded
with url: /upload/storage/v1/b//o?uploadType=multipart
(Caused by SSLError(SSLEOFError(8,
'[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1032)')))
```

The failures are infrequent, but they have occurred across multiple Cloud Run instances and independent upload jobs. They persisted when we stopped sharing clients between application worker threads and instead created and closed a client for every upload.

While investigating why client isolation did not eliminate concurrent use of a transport, we found that 3.11.0 introduced App-centric Observability bucket metadata enrichment. On the first blob operation for a bucket, `create_trace_span_helper()` calls `BucketMetadataCache.get_or_queue_fetch()`. A daemon thread then calls:

```python
self._client.get_bucket(bucket_name, timeout=10.0)
```

`BucketMetadataCache` retains the same `Client` used by the foreground operation. Consequently, the metadata request and multipart upload can call the same `google.auth.transport.requests.AuthorizedSession` concurrently. This happens even when application code guarantees that only one worker thread uses a client.

When clients are short-lived, the cache begins empty for every upload. Cloud Storage request metrics then show approximately one `GetBucketMetadata` request for every `WriteObject` request. In other words, creating a fresh client per upload increases rather than removes this hidden concurrency.

This report does **not** claim that the shared session is conclusively the source of every TLS EOF. The evidence establishes the unexpected concurrent transport use and a strong version/timing correlation. We would like maintainer guidance on whether concurrent use of one `AuthorizedSession` by the cache and a foreground transfer is intended and supported.

Related issue #17650 describes the same cache's unexpected `storage.buckets.get` calls and IAM/audit-log impact. This report focuses on transport sharing and reliability during object transfers.

### Expected Behavior

An object upload performed with an otherwise isolated `storage.Client` should not concurrently use that client's HTTP transport from an internal daemon thread.

If background bucket metadata collection is required, it should use an independently owned transport, be enabled only when its telemetry is used, or be configurable per client. A failure or retry storm in optional metadata enrichment should not overlap or interfere with the foreground transfer's transport and retry window.

### Actual Behavior

The first blob operation on a cache miss starts `_fetch_background` in a daemon thread. That thread and the foreground upload use the same client and `AuthorizedSession` concurrently.

Operationally, multipart uploads intermittently fail after the full retry deadline with `SSLEOFError: UNEXPECTED_EOF_WHILE_READING`. The first observed failures appeared in the first deployment containing `google-cloud-storage` 3.13.0; earlier retained logs using 3.10.1 did not contain this failure signature. Changing application-level client ownership from thread-local clients to one client per upload did not remove the failures.

We are using a rollback to 3.10.1 as a controlled diagnostic because it predates `BucketMetadataCache`. We will update the upstream issue with the result after sufficient comparable traffic.

## API client name and version

`google-cloud-storage` 3.13.0

The relevant behavior was introduced in 3.11.0 and remains in 3.13.1. Version 3.13.1 includes the process-wide `DISABLE_GCS_PYTHON_CLIENT_OTEL_BUCKET_METADATA` environment variable, but it does not isolate the background request's transport when metadata enrichment is enabled.

## Reproduction steps: code

The intermittent TLS failure depends on a connection being closed at an unfortunate point and is not deterministic. The following complete diagnostic instead reproduces the prerequisite hidden concurrency: a metadata daemon and multipart upload entering the same `AuthorizedSession.request()` method at the same time.

It requires Application Default Credentials and an existing test bucket supplied through `GCS_BUCKET`. It uploads and deletes one temporary object.

file: `main.py`

```python
import os
import threading
import uuid

# Ensure the behavior under investigation is enabled before importing storage.
os.environ.pop("DISABLE_GCS_PYTHON_CLIENT_OTEL_BUCKET_METADATA", None)

from google.cloud import storage

bucket_name = os.environ["GCS_BUCKET"]
client = storage.Client()
transport = client._http
original_request = transport.request

metadata_entered = threading.Event()
upload_entered = threading.Event()
metadata_finished = threading.Event()

def instrumented_request(method, url, *args, **kwargs):
method = method.upper()
is_metadata = method == "GET" and "/storage/v1/b/" in url
is_multipart_upload = "uploadType=multipart" in url

if is_metadata:
print(
"metadata request entered:",
f"thread={threading.current_thread().name}",
f"session_id={id(transport)}",
)
metadata_entered.set()
upload_entered.wait(timeout=30.0)
elif is_multipart_upload:
metadata_entered.wait(timeout=30.0)
print(
"upload request entered:",
f"thread={threading.current_thread().name}",
f"session_id={id(transport)}",
f"metadata_already_entered={metadata_entered.is_set()}",
)
upload_entered.set()

try:
return original_request(method, url, *args, **kwargs)
finally:
if is_metadata:
metadata_finished.set()

transport.request = instrumented_request
blob = client.bucket(bucket_name).blob(f"metadata-cache-repro/{uuid.uuid4()}.txt")

try:
blob.upload_from_string(b"test", content_type="text/plain")
metadata_finished.wait(timeout=60.0)
finally:
transport.request = original_request
try:
blob.delete()
finally:
client.close()
```

## Reproduction steps: supporting files

None.

## Reproduction steps: actual results

With `google-cloud-storage` 3.13.0, the diagnostic shows the metadata request entering from `_fetch_background` and the multipart upload entering from the application thread with the same session identifier:

```text
metadata request entered: thread=Thread-1 (_fetch_background) session_id=
upload request entered: thread=MainThread session_id= metadata_already_entered=True
```

The two waits make the overlap deterministic: the metadata thread does not continue until the upload has entered the same transport.

In normal, uninstrumented operation, most uploads succeed. A controlled run of 64 concurrent uploads also completed successfully, confirming that a connection-level failure is needed to expose the intermittent problem. When the failure occurs in the deployed service, the final result after retries is the `SSLEOFError` traceback shown in the summary.

With 3.10.1, there is no metadata daemon and the diagnostic prints only the upload request with `metadata_already_entered=False`.

## Reproduction steps: expected results

The upload should be the only request using its `AuthorizedSession`, or any automatic background metadata request should use a separate transport:

```text
upload request entered: thread=MainThread session_id= metadata_already_entered=False
```

## OS & version + platform

Debian Linux container on Google Cloud Run.

We observed the deployed failures with OpenSSL reporting `_ssl.c:1032`. Local inspection and controlled testing used Linux with OpenSSL 3.5.6.

## Python environment

Python 3.13.15 in the affected Cloud Run revision. Controlled local testing used Python 3.13.14.

## Python dependencies

```text
google-api-core==2.33.0
google-auth==2.56.2
google-cloud-core==2.6.0
google-cloud-storage==3.13.0
google-resumable-media==2.10.0
requests==2.34.2
urllib3==2.7.0
cryptography==50.0.0
```

## Additional context

### Version boundary and call chain

- 3.10.1 does not contain `BucketMetadataCache`.
- 3.11.0 added "Enhance Otel Span Attributes with BucketId and Location details for every Bucket/Blob operation."
- On a cache miss, `get_or_queue_fetch()` starts a daemon `threading.Thread`.
- `_fetch_background()` calls `self._client.get_bucket(bucket_name, timeout=10.0)`.
- `get_bucket()` retains its default `DEFAULT_RETRY`; the 10-second request timeout does not by itself disable the normal retry policy.
- Both requests therefore use the same client's HTTP transport. Application-level thread-local or per-operation client ownership does not prevent this internal sharing.
- 3.13.1 still contains this implementation. Its environment-variable opt-out avoids the cache globally, but there is no per-client ownership choice and no transport isolation while the feature is enabled.

Relevant source:

- [3.11.0 changelog entry](https://github.com/googleapis/google-cloud-python/blob/google-cloud-storage-v3.11.0/packages/google-cloud-storage/CHANGELOG.md#L49-L55)
- [`BucketMetadataCache.get_or_queue_fetch()` in 3.13.1](https://github.com/googleapis/google-cloud-python/blob/google-cloud-storage-v3.13.1/packages/google-cloud-storage/google/cloud/storage/_bucket_metadata_cache.py#L45-L65)
- [`BucketMetadataCache._fetch_background()` in 3.13.1](https://github.com/googleapis/google-cloud-python/blob/google-cloud-storage-v3.13.1/packages/google-cloud-storage/google/cloud/storage/_bucket_metadata_cache.py#L94-L113)
- [Related issue #17650](https://github.com/googleapis/google-cloud-python/issues/17650)

### Other causes investigated

During affected periods we did not find corresponding CPU exhaustion, memory pressure, Cloud Run request-concurrency spikes, outbound packet/byte throttling, instance lifecycle events, or a Cloud NAT path. Failures occurred on different instances rather than one persistently unhealthy process. Cloud Storage metrics showed successful writes surrounding a small number of `UNAVAILABLE` or `CANCELLED` requests.

urllib3 2.7.0 removes and closes a connection after an SSL exception before a retry, so the library's retries are not repeatedly selecting the same known-failed pooled socket. Recreating the whole `storage.Client` also did not improve recovery, which led us to inspect what else a new client starts during its first operation.

### Questions and requested change

1. Is concurrent use of one `AuthorizedSession` by `_fetch_background()` and a foreground upload an intended and supported client contract?
2. Could `BucketMetadataCache` use a separately owned client/transport, or collect metadata synchronously before a transfer begins, so optional enrichment cannot overlap the transfer on one session?
3. Could the cache be disabled per `Client` through a public constructor option? The environment variable in 3.13.1 is useful but process-wide and unavailable to callers that need different behavior for different clients.
4. Could a regression test exercise uploads while the metadata request encounters delayed responses, connection resets, and TLS EOFs, verifying that the foreground transfer remains isolated?

We can test a proposed patch or provide additional sanitized timing and request-level diagnostics.