[C++][Python] GcsFileSystem: can't access Requester Pays buckets
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Cannot access GCS Requester Pays buckets despite setting `project_id`. There is no way to specify a billing/quota project for requests — any access attempt returns `Bucket is a requester pays bucket but no user project provided`.
### Reproduction
Python — all of the following fail with 400 UserProjectMissing:
```
import pyarrow.fs as pafs
# Attempt 1: project_id parameter
fs = pafs.GcsFileSystem(project_id="my-billing-project")
fs.get_file_info("requester-pays-bucket/file.parquet") # 400 UserProjectMissing
# Attempt 2: access token with project
fs = pafs.GcsFileSystem(
access_token="",
credential_token_expiration=expiry,
project_id="my-billing-project",
)
fs.get_file_info("requester-pays-bucket/file.parquet") # 400 UserProjectMissing
```
Environment variables — all ignored by Arrow, but work with gsutil/gcloud:
```
# Set quota project via ADC
gcloud auth application-default set-quota-project my-billing-project
# Or via environment variable
export GOOGLE_CLOUD_QUOTA_PROJECT=my-billing-project
# gsutil works fine with either of the above:
gsutil ls gs://requester-pays-bucket/
# pyarrow does not — the env/ADC quota project is never passed through to the GCS client
```
### Reference:
[Google ccp client requester pays example](https://docs.cloud.google.com/storage/docs/using-requester-pays#using)
[Google AIP for auth]( https://google.aip.dev/auth/4110)
### Component(s)
C++, Python
Contributor guide
Assessment
This issue has not been assessed yet.