apache / apache/arrow-rs-object-store
Support for GCS requester pays
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm working with a requester pays dataset within Google Cloud Storage, I'm using upstream packages (obstore + async-tiff) to access the data, but there's no support for requester pays buckets. These packages have issues mentioning this missing [feature](https://github.com/developmentseed/obstore/issues/546).
**Describe the solution you'd like**
Add the documented behaviour of the google cloud storage requester pays:
- Add the x-goog-user-project header to requests to google cloud storage when configured (meaning this project is charged for access to the bucket)
- Add the userProject query parameter to signed urls
- Add x-goog-user-project header to each request for multi-part requests, as each request is billed separately
Docs:
- [Access requester pays](https://docs.cloud.google.com/storage/docs/using-requester-pays#using)
- [Multi-request operations](https://docs.cloud.google.com/storage/docs/requester-pays#multi-request)
- `x-goog-user-project` - https://docs.cloud.google.com/storage/docs/xml-api/reference-headers#xgooguserproject
- `userProject` - https://docs.cloud.google.com/storage/docs/xml-api/reference-headers#userProject
**Describe alternatives you've considered**
N/A
**Additional context**
I'm working on this already, but I'm new to rust, so taking a while to review and validate. WIP [here](https://github.com/apache/arrow-rs-object-store/compare/main...jakenotjay:arrow-rs-object-store:feature/gcp-requester-pays).
Contributor guide
Research direction
Start with the linked Google Cloud Storage requester-pays documentation and the WIP comparison branch. Verify that configured requests and each multipart request send x-goog-user-project, and that signed URLs include userProject; done means these behaviors are covered and validated against the documented requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100