basalt-26.3: roachtest: apt_problem failed

roachtest.apt_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21475188?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21475188?buildTab=artifacts#/db-console/mixed-version-cypress) on basalt-26.3 @ [ff8f3ca43cd223b5da6e313b03e4014474aa9b8b](https://github.com/cockroachdb/cockroach/commits/ff8f3ca43cd223b5da6e313b03e4014474aa9b8b):

Failed with:

```
test db-console/mixed-version-cypress failed: (assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/db_console_cypress.go:145
pkg/cmd/roachtest/tests/db_console_cypress.go:86
pkg/cmd/roachtest/tests/db_console_cypress.go:304
pkg/cmd/roachtest/roachtestutil/mixedversion/steps.go:468
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:268
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:819
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:267
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:242
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:162
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
full command output in run_062644.901746363_n5_sudo-aptget-update-s.log: error persisted after 3 attempts: TRANSIENT_ERROR(apt_problem): exit status 100
(1) attached stack trace
-- stack trace:
| main.(*roachprodCluster).RunE
| pkg/cmd/roachtest/cluster.go:3426
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*dbConsoleCypressTest).installDeps
| pkg/cmd/roachtest/tests/db_console_cypress.go:145
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*dbConsoleCypressTest).SetupTest
| pkg/cmd/roachtest/tests/db_console_cypress.go:86
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runDbConsoleCypressMixedVersions.func1
| pkg/cmd/roachtest/tests/db_console_cypress.go:304
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.runHookStep.Run
| pkg/cmd/roachtest/roachtestutil/mixedversion/steps.go:468
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep.func2
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:268
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.panicAsError
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:819
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:267
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:242
| [...repeated from below...]
Wraps: (2) full command output in run_062644.901746363_n5_sudo-aptget-update-s.log
Wraps: (3) forced error mark
| "error occurred after retries"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.runWithMaybeRetry
| pkg/roachprod/install/cluster_synced.go:244
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE.func1.1
| pkg/roachprod/install/cluster_synced.go:2715
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (5) error persisted after 3 attempts
Wraps: (6) secondary error attachment
| TRANSIENT_ERROR(apt_problem): exit status 100
| (1) Node 5. Command with error:
| | ``````
| | sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2 libxtst6 xauth xvfb
| | ``````
| | ... s_1.0.5-0ubuntu2_all.deb 503 Service Unavailable [IP: 18.190.157.201 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/x/xfonts-utils/xfonts-utils_7.7%2b6build2_arm64.deb 503 Service Unavailable [IP: 18.190.157.201 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/x/xfonts-base/xfonts-base_1.0.5_all.deb 503 Service Unavailable [IP: 18.190.157.201 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libglvnd0_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.190.157.201 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libglx0_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.190.157.201 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libgl1_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.190.157.201 80]
| | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
| Wraps: (2) TRANSIENT_ERROR(apt_problem)
| Wraps: (3) exit status 100
| Error types: (1) *hintdetail.withDetail (2) errors.TransientError (3) *exec.ExitError
Wraps: (7) secondary error attachment
| TRANSIENT_ERROR(apt_problem): exit status 100
| (1) Node 5. Command with error:
| | ``````
| | sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2 libxtst6 xauth xvfb
| | ``````
| | ... s_1.0.5-0ubuntu2_all.deb 503 Service Unavailable [IP: 18.216.209.118 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/x/xfonts-utils/xfonts-utils_7.7%2b6build2_arm64.deb 503 Service Unavailable [IP: 18.216.209.118 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/x/xfonts-base/xfonts-base_1.0.5_all.deb 503 Service Unavailable [IP: 18.216.209.118 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libglvnd0_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.216.209.118 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libglx0_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.216.209.118 80]
| | E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libgl1_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.216.209.118 80]
| | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
| Wraps: (2) TRANSIENT_ERROR(apt_problem)
| Wraps: (3) exit status 100
| Error types: (1) *hintdetail.withDetail (2) errors.TransientError (3) *exec.ExitError
Wraps: (8) Node 5. Command with error:
| ``````
| sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2 libxtst6 xauth xvfb
| ``````
| ... codings_1.0.5-0ubuntu2_all.deb 503 Service Unavailable [IP: 18.219.209.11 80]
| E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/x/xfonts-utils/xfonts-utils_7.7%2b6build2_arm64.deb 503 Service Unavailable [IP: 18.219.209.11 80]
| E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/x/xfonts-base/xfonts-base_1.0.5_all.deb 503 Service Unavailable [IP: 18.219.209.11 80]
| E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libglvnd0_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.219.209.11 80]
| E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libglx0_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.219.209.11 80]
| E: Failed to fetch http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports/pool/main/libg/libglvnd/libgl1_1.4.0-1_arm64.deb 503 Service Unavailable [IP: 18.219.209.11 80]
| E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Wraps: (9) TRANSIENT_ERROR(apt_problem)
Wraps: (10) exit status 100
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *markers.withMark (4) *withstack.withStack (5) *errutil.withPrefix (6) *secondary.withSecondaryError (7) *secondary.withSecondaryError (8) *hintdetail.withDetail (9) errors.TransientError (10) *exec.ExitError
Test: db-console/mixed-version-cypress
(require.go:1401).NoError: FailNow called
(mixedversion.go:926).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/db-console/mixed-version-cypress/cpu_arch=arm64/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21475188-1787204533-18-n5cpu4-0001 | 18.227.95.207 | 10.12.45.170 |
| teamcity-21475188-1787204533-18-n5cpu4-0002 | 16.59.212.184 | 10.12.37.4 |
| teamcity-21475188-1787204533-18-n5cpu4-0003 | 13.58.199.241 | 10.12.32.130 |
| teamcity-21475188-1787204533-18-n5cpu4-0004 | 18.227.98.116 | 10.12.40.122 |
| teamcity-21475188-1787204533-18-n5cpu4-0005 | 13.58.153.232 | 10.12.32.31 |

Parameters:
- arch=arm64
- cloud=aws
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=system-only
- mvtVersions=v25.3.4 → v25.4.2 → v26.1.2 → v26.2.1 → v26.3.1-dev-ff8f3ca43cd223b5da6e313b03e4014474aa9b8b (basalt-26.3)
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21475188/db-console-mixed-version-cypress/1787207126066/1787207873720)

_Logs were not uploaded to Datadog_

Same failure on other branches

- #173563 release-26.2.6-rc: roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.6-rc]
- #173550 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.23-rc]
- #173460 release-26.3.1-rc: roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3.1-rc]
- #172603 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.8-rc]
- #172566 release-26.3.0-rc: roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3.0-rc]
- #172511 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.22-rc]
- #171789 release-26.3: roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3]
- #171651 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.12-rc]
- #171285 roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.5-rc]
- #171154 release-26.2.2-rc: roachtest: apt_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.2-rc]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*apt_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66898

Read pkg/cmd/roachtest/tests/db_console_cypress.go, especially installDeps and the mixed-version Cypress setup, then inspect run_062644.901746363_n5_sudo-aptget-update-s.log. Reproduce the arm64 AWS failure if possible and determine the project change needed so the dependency installation no longer fails on the reported Ubuntu package-server errors; done means the roachtest passes reliably.

Written by the indexing model from the issue text.