cockroachdb / cockroachdb/cockroach

release-26.2.7-rc: roachtest: cluster_creation failed

Open
#174,642 328 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2.7-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21497652?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21497652?buildTab=artifacts#/acceptance/gossip/restart-node-one) on release-26.2.7-rc @ [b1582e0a72ed7f3813a4346a2ddce2d9a29c30c2](https://github.com/cockroachdb/cockroach/commits/b1582e0a72ed7f3813a4346a2ddce2d9a29c30c2):

```
test acceptance/gossip/restart-node-one failed: (test_runner.go:1027).func4: failed to retrieve authorized keys from gcloud: read SSH keys from gs://roachprod-ssh-keys-crl-e2e-infra/ssh-keys: googleapi: got HTTP response code 403 with body: AccessDeniedAccess denied.teamcity-nightly@cockroach-ephemeral.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource '//storage.googleapis.com/projects/_/buckets/roachprod-ssh-keys-crl-e2e-infra/objects/ssh-keys' (or it may not exist). [owner=test-eng]
```

Parameters:
- cloud=aws
- coverageBuild=false
- cpu=4
- diskCount=0
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #174641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.17-rc]
- #174639 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.24-rc]
- #174597 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174596 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174595 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174594 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174593 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174592 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174591 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174589 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67839

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the failure at test_runner.go:1027 for acceptance/gossip/restart-node-one. Review the linked TeamCity logs and artifacts, then investigate the reported 403 while reading the GCS ssh-keys object; done means the cluster_creation roachtest no longer fails for this access error across the affected branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, gcp, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.