apache / apache/arrow

[C++][Python] GcsFileSystem: can't access Requester Pays buckets

Open
#49,395 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Cannot access GCS Requester Pays buckets despite setting `project_id`. There is no way to specify a billing/quota project for requests — any access attempt returns `Bucket is a requester pays bucket but no user project provided`.

### Reproduction

Python — all of the following fail with 400 UserProjectMissing:
```
import pyarrow.fs as pafs

# Attempt 1: project_id parameter
fs = pafs.GcsFileSystem(project_id="my-billing-project")
fs.get_file_info("requester-pays-bucket/file.parquet") # 400 UserProjectMissing

# Attempt 2: access token with project
fs = pafs.GcsFileSystem(
access_token="",
credential_token_expiration=expiry,
project_id="my-billing-project",
)
fs.get_file_info("requester-pays-bucket/file.parquet") # 400 UserProjectMissing
```
Environment variables — all ignored by Arrow, but work with gsutil/gcloud:
```
# Set quota project via ADC
gcloud auth application-default set-quota-project my-billing-project

# Or via environment variable
export GOOGLE_CLOUD_QUOTA_PROJECT=my-billing-project

# gsutil works fine with either of the above:
gsutil ls gs://requester-pays-bucket/

# pyarrow does not — the env/ADC quota project is never passed through to the GCS client
```

### Reference:
[Google ccp client requester pays example](https://docs.cloud.google.com/storage/docs/using-requester-pays#using)
[Google AIP for auth]( https://google.aip.dev/auth/4110)

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.