cockroachdb / cockroachdb/cockroach

roachtest: apt_problem failed

Open
#147,482 0 comments 0 reactions 0 assignees View on GitHub
branch-release-24.3.14-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.apt_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/19807840?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/19807840?buildTab=artifacts#/jepsen/bank/start-kill-2) on release-24.3.14-rc @ [ea12d5729d66c53565948efdb060b2244f06afe7](https://github.com/cockroachdb/cockroach/commits/ea12d5729d66c53565948efdb060b2244f06afe7):

```
test jepsen/bank/start-kill-2 failed: (cluster.go:2474).Run: full command output in run_094158.147641613_n1-6_sh-c-sudo-DEBIANFRON.log: error persisted after 3 attempts: TRANSIENT_ERROR(apt_problem): exit status 100
test artifacts and logs in: /artifacts/jepsen/bank/start-kill-2/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=true
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-19807840/jepsen-bank-start-kill-2/1748511601730/1748511920732)

Same failure on other branches

- #147481 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.26-rc]
- #139470 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1]
- #133528 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3]
- #133525 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
- #132737 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
- #132063 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc]
- #126897 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]
- #124052 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*apt_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-51075

Contributor guide

Open the contributing guide

Research direction

Start with the run_094158.147641613_n1-6_sh-c-sudo-DEBIANFRON.log artifact and the roachtest README, then inspect cluster.go:2474 and the jepsen/bank/start-kill-2 entry point. Determine why apt exits with status 100 across the related failures and verify the fix against this roachtest.

Written by the indexing model from the issue text.

Assessment

Tech stack
debian, gcp, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.