googleapis / googleapis/google-cloud-python

google-cloud-storage: BucketMetadataCache (ACO) triggers storage.buckets.get on object-only operations; no supported opt-out

Ouverte
#17,650 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
5.4k
Forks
1.8k
Merge moyen
3 j 4 h
PR mergées (30 j)
122

Description

## Environment details

- **Programming language**: Python 3.11
- **Package**: `google-cloud-storage`
- **Versions observed**: `3.11.0` and `3.12.0` (the `BucketMetadataCache` / ACO code path is present in both)

## Summary

`Client.__init__` unconditionally instantiates a `BucketMetadataCache` ("App-centric Observability" / ACO). As a side effect, **ordinary object-level operations** (`list_blobs`, `download_*`, `upload_*`, etc.) now trigger a **background `Client.get_bucket()` call**, i.e. the REST `storage.buckets.get` permission — even though the application code never requests bucket metadata and the operation itself only needs `storage.objects.*`.

For principals granted **object-only roles** (`roles/storage.objectViewer`, `roles/storage.objectAdmin`) — which intentionally exclude `storage.buckets.get` — this produces a continuous stream of **denied `storage.buckets.get` entries in Cloud Audit Logs (data_access, severity ERROR)**, one per bucket per client instance.

The failure is swallowed internally (data operations still succeed), so it is not a functional outage — but it is:
- **Alarming, high-volume audit-log noise** (ERROR severity, "lots per second" under fan-out workloads that create many short-lived clients), and
- **Pressure to over-grant IAM** — teams are asked to add `storage.buckets.get` (e.g. via `roles/storage.bucketViewer`/`legacyBucketReader`) purely to satisfy a telemetry probe the application does not use, across *every* bucket the service account ever touches. This directly conflicts with least-privilege.

There is **no supported way to disable it** — no client option and no environment variable (the only observability env var, `ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES`, is a separate opt-in for OpenTelemetry tracing). The only workaround is mutating the private attribute `client._bucket_metadata_cache = None`, which is fragile and fails open on library upgrades.

## Steps to reproduce

Object-only operations cause a background `get_bucket()` (→ `storage.buckets.get`). The repro below is **fully offline and needs no credentials or real bucket** — it monkeypatches `get_bucket` only to *observe that it is invoked and by whom*:

```python
import time
import traceback

from google.auth.credentials import AnonymousCredentials
from google.cloud import storage

calls = []
def traced_get_bucket(self, bucket_name, *a, **k):
if not calls:
print("get_bucket() caller:\n" + "".join(traceback.format_stack()[-3:-1]))
calls.append(bucket_name)
raise RuntimeError("short-circuit (offline repro)") # swallowed by ACO bg thread
storage.Client.get_bucket = traced_get_bucket

client = storage.Client(project="proof", credentials=AnonymousCredentials())
# Optional workaround under test: client._bucket_metadata_cache = None
try:
list(client.list_blobs("some-bucket", prefix="x/", max_results=1))
except Exception:
pass # objects.list fails offline; irrelevant to the point
time.sleep(1.0) # let the ACO daemon thread run

print(f"google-cloud-storage {storage.__version__}: "
f"get_bucket invoked {len(calls)} time(s) by one list_blobs() call")
```

### Actual output

```
get_bucket() caller:
File ".../google/cloud/storage/_bucket_metadata_cache.py", line 98, in _fetch_background
bucket = self._client.get_bucket(bucket_name, timeout=10.0)
google-cloud-storage 3.11.0: get_bucket invoked 1 time(s) by one list_blobs() call
```

With `client._bucket_metadata_cache = None` uncommented, the count is `0`.

### Root cause (call chain)

1. `_helpers.create_trace_span_helper` wraps every operation. For any span **not** in `{Storage.Client.getBucket, Storage.Client.lookupBucket, Storage.Bucket.reload, Storage.Bucket.exists}` (i.e. object ops like `listBlobs`), it calls `client._bucket_metadata_cache.get_or_queue_fetch(bucket_name)`.
2. On a cache miss, `get_or_queue_fetch` spawns a daemon thread → `_bucket_metadata_cache._fetch_background`.
3. `_fetch_background` calls `self._client.get_bucket(bucket_name, timeout=10.0)` → REST `storage.buckets.get`.
4. On `Forbidden`, it caches a fallback and returns; the object op proceeds. Every fresh client re-probes on first touch of each bucket.

## Expected behavior

Object-level operations should not require `storage.buckets.get`. This is a long-standing, explicitly-recognized contract for this library — cf. googleapis/python-storage #528 ("Redesign of API so it does not need `storage.buckets.get` access for every single operation") and #1225 ("Need to be able to download objects without `storage.buckets.get` permission same as gsutil"). `gsutil`/`gcloud storage` operate on objects without `buckets.get`, and object-only IAM roles deliberately omit it.

At minimum, ACO's background metadata probe should be **opt-out via a supported mechanism**, and ideally should not fire for principals/operations that don't need bucket metadata.

## Requested change

A **supported opt-out**, e.g. one or more of:

1. A `Client` constructor option (e.g. `enable_bucket_metadata_cache: bool = True`), and/or
2. An environment variable mirroring the existing `ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES` pattern (e.g. `DISABLE_GCS_PYTHON_CLIENT_BUCKET_METADATA_CACHE`), and/or
3. Making the probe lazy/suppressible so a `Forbidden` result is cached **once per client** without re-emitting audit-log errors, and documenting that ACO requires `storage.buckets.get`.

Any of these lets callers using object-only roles avoid the audit-log noise **without granting `storage.buckets.get`**, and gives a stable alternative to mutating the private `_bucket_metadata_cache` attribute.

Happy to open a PR for option 1 or 2 if a maintainer confirms the preferred shape.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Start with Client.__init__, _helpers.create_trace_span_helper, and google/cloud/storage/_bucket_metadata_cache.py to trace when object operations queue metadata fetches. Compare the requested opt-out approaches, then add coverage showing the selected supported mechanism prevents the background get_bucket call while object operations remain usable.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
api, cloud
Type d'issue
Fonctionnalité
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
52/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.