GoogleCloudPlatform / GoogleCloudPlatform/testgrid

Tabulator runs into GCS object rate-limiting right at startup

Open
#1,036 5 comments 0 reactions 1 assignee Claimed by @chases2 View on GitHub
bug help wanted kind/oncall-hotlist
Dominant language
Go
Stars
205
Forks
73
Avg merge
1d 9h
Merged PRs (30d)
6

Description

When the Tabulator first boots, it will try to do a lot of writes quickly. When several of those [writes are on the same object](https://stackoverflow.com/questions/32149622/rate-limiting-in-google-cloud-storage/32150168#32150168), we get blocks of these errors (about 6 or so on the same object) for a little while:

```
jsonPayload: {
config: "gs://k8s-testgrid/config"
dashboard: "sig-node-containerd"
file: "pkg/tabulator/tabstate.go:273"
func: "github.com/GoogleCloudPlatform/testgrid/pkg/tabulator.Update.func5"
group: "ci-cos-containerd-node-e2e"
level: "error"
msg: "write: client.Upload(gs://k8s-testgrid/tabs/sig-node-containerd/image-validation-node-e2e): close: googleapi: Error 429: The rate of change requests to the object k8s-testgrid/tabs/sig-node-containerd/image-validation-node-e2e exceeds the rate limit. Please reduce the rate of create, update, and delete requests., rateLimitExceeded"
tab: "image-validation-node-e2e"
}
```

Then things tend to stabilize, but this burst of errors occurs pretty regularly whenever the tabulator starts up.

Additional syncing may be needed to prevent two (or six!) writing goroutines from trying to update exactly the same file.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.