microsoft / microsoft/retina

GCP Bucket Support through S3 compatible - SignatureDoesNotMatch 403 Error

Open
#694 0 comments 0 reactions 0 assignees View on GitHub
area/captures type/bug
Dominant language
Go
Stars
3.2k
Forks
304
Avg merge
1d 16h
Merged PRs (30d)
79

Description

**Describe the bug**
As S3 compatible buckets are supported, I expected that GCP Buckets would also work. [Tutorial](https://medium.com/google-cloud/use-case-2-aws-s3-compatible-interoperable-google-cloud-storage-b23ad7063bff#:~:text=AWS%20S3%2DCompatible%20and%20Interoperable,Simple%20Storage%20Service%20(S3).)

But uppon implementing, we receive the following error:

```logs
level=error caller=outputlocation/s3.go:111 msg="Couldn't upload file" srcFilePath=/tmp/capture-node-blobupload-gke-nodepool-1-sdfa234-4445-20240902052745UTC.tar.gz bucketName=tmp-packet-captures objectKey=retina/captures/tmp/capture-node-blobupload-gke-nodepool-1-sdfa234-4445-20240902052745UTC.tar.gz error="failed to upload file to S3: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: Access denied."
```

**To Reproduce**
Version v0.0.14 through Helm Chart and CRD

1. Follow all the steps on tutorial, configure the HMAC key, region and secret
2. Add IAM permission for Bucket write access
3. Initialize Capture

**Expected behavior**
Upload to GCP Bucket should just work.

**Platform (please complete the following information):**

- OS: Linux
- Kubernetes Version: v1.28.11
- Host: GCP
- Retina Version: V0.0.14 Through Helm Chart

**Additional context**
I found this [Stackoverflow link](https://stackoverflow.com/questions/73717477/gcp-cloud-storage-golang-aws-sdk2-upload-file-with-s3-interoperability-creds), seems like the issue is due to the v2 library: `According to https://github.com/aws/aws-sdk-go-v2/issues/1816 the v2 SDK doesn't support GCP GCS.`

Not sure how to fix, but would be a great plus if it just worked.

Configuration file:

```yaml

apiVersion: retina.sh/v1alpha1
kind: Capture
metadata:
name: capture-node-blobupload
spec:
captureConfiguration:
captureOption:
duration: 10s
captureTarget:
nodeSelector:
matchExpressions:
- {
key: kubernetes.io/hostname,
operator: In,
values: [gke-nodepool-1-sdfa234-4445],
}
outputConfiguration:
hostPath: "/tmp/retina"
s3Upload:
endpoint: https://storage.googleapis.com
bucket: tmp-packet-captures
region: auto # I also tried us-central1 for clear of mind.
path: retina/captures
secretName: capture-s3-upload-secret
---
apiVersion: v1
kind: Secret
metadata:
name: capture-s3-upload-secret
stringData:
s3-access-key-id:
s3-secret-access-key:
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.