vectordotdev / vectordotdev/vector
Add support in gcp_cloud_storage sink to use signed url to upload files
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
GCS supports signed urls for file upload:
https://cloud.google.com/storage/docs/access-control/signed-urls
This feature request is for running vector in a less controlled environment where we don't want to provide it with a GCP service account (SA) that has the write permission to the GCS bucket.
When the sink needs to upload a file to GCS, it will first query another Service "A" (better controlled) with the "filename" to be uploaded (using query param with GET or request body with POST), and that service will return the signed url. The sink can then use the signed url to upload the file.
Service "A" may need authenticate the sink using a GCP service account. The sink won't need an SA that has write permission to the GCS bucket. It only needs an SA that can talk to Service A.
Only Service A will have access to an SA that has GCS bucket write permission. It also has the mapping of sink SA and corresponding bucket/folder to store the files from that sink. Multiple sinks with different SA can query the same Service A.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the gcp_cloud_storage sink and reviewing Google Cloud Storage signed URL documentation. Define how the sink requests a URL from Service A, authenticates that request, and uses the returned URL for uploads; done means files upload without granting the sink service account bucket-write access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, rust
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100