When using gcp storage, multiple "Error 403: Access denied., forbidden" in "gcp.removeInProgressMetadata"
- Dominant language
- Go
- Stars
- 4.8k
- Forks
- 542
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 2
Description
**Describe the bug**
Using gcp storage, very frequently I get this "Error 403: Access denied., forbidden" from GCS, always in the "gcp.removeInProgressMetadata" step.
Looking at the code, it seems this is using "object.update" to clear object metadata.
The service account has "storageAdmin" permissions, calling the API directly in the exact same file to remove the metadata works every time.
**Error Message**
If applicable add error message to help explain your problem or console output enclose in triple back-ticks eg,
```console
{"http-method":"GET","http-path":"/cloud.google.com/go/secretmanager/@v/v1.14.1.mod","kind":"Internal Server Error","level":"error","module":"","msg":"googleapi: Error 403: Access denied., forbidden","operation":"download.VersionModuleHandler","ops":["download.VersionModuleHandler","pool.GoMod","protocol.GoMod","protocol.processDownload","gcslock.Stash","stash.Pool","stasher.Stash","gcp.save","gcp.removeInProgressMetadata"],"request-id":"13cb4103-3c37-423a-b5d6-f7a7df1bbdf5","time":"2024-12-12T18:21:20Z","version":""}
```
**To Reproduce**
It happens every time on our big monorepo.
**Expected behavior**
No error as the service account have the correct permission.
**Environment (please complete the following information):**
- OS: Linux 64bit
- Go version : 1.23.1
- Proxy version : 0.15.4
- Storage gcp :
**Additional context**
It looks like some kind of locking problem, maybe it is setting the metadata but the file isn't really there yet as GCS is eventually consistent? I saw some terraform threads about this message being really "object not found" instead of a permission error.
Contributor guide
Assessment
This issue has not been assessed yet.