roachtest: install_flake): error persisted after 3 attempts: TRANSIENT_ERROR(apt_problem failed

roachtest.install_flake): error persisted after 3 attempts: TRANSIENT_ERROR(apt_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20920785?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20920785?buildTab=artifacts#/db-console/mixed-version-cypress) on release-26.1 @ [697c33f7d0997004dca91195440a97a833258eaf](https://github.com/cockroachdb/cockroach/commits/697c33f7d0997004dca91195440a97a833258eaf):

Failed with:

```
test db-console/mixed-version-cypress failed: (assertions.go:369).Fail:
Error Trace: pkg/cmd/roachtest/tests/db_console_cypress.go:159
pkg/cmd/roachtest/tests/db_console_cypress.go:99
pkg/cmd/roachtest/tests/db_console_cypress.go:259
pkg/cmd/roachtest/roachtestutil/mixedversion/steps.go:468
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:268
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:819
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:267
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:242
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:162
src/runtime/asm_amd64.s:1693
Error: Received unexpected error:
cluster.Install: retryable infrastructure error: could not install [docker]: TRANSIENT_ERROR(install_flake): error persisted after 3 attempts: TRANSIENT_ERROR(apt_problem): exit status 100
(1) attached stack trace
-- stack trace:
| main.(*clusterImpl).Install
| pkg/cmd/roachtest/cluster.go:2679
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*dbConsoleCypressTest).buildDockerImage
| pkg/cmd/roachtest/tests/db_console_cypress.go:159
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*dbConsoleCypressTest).SetupTest
| pkg/cmd/roachtest/tests/db_console_cypress.go:99
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runDbConsoleCypressMixedVersions.func1
| pkg/cmd/roachtest/tests/db_console_cypress.go:259
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.runHookStep.Run
| pkg/cmd/roachtest/roachtestutil/mixedversion/steps.go:468
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep.func2
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:268
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.panicAsError
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:819
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:267
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:242
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).run.func3
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:162
Wraps: (2) cluster.Install
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod.Install.func1
| pkg/roachprod/roachprod.go:1088
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1
| pkg/util/retry/retry.go:337
| github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
| pkg/util/retry/retry.go:291
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
| pkg/util/retry/retry.go:336
| github.com/cockroachdb/cockroach/pkg/roachprod.Install
| pkg/roachprod/roachprod.go:1083
| main.(*clusterImpl).Install
| pkg/cmd/roachtest/cluster.go:2679
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*dbConsoleCypressTest).buildDockerImage
| pkg/cmd/roachtest/tests/db_console_cypress.go:159
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*dbConsoleCypressTest).SetupTest
| pkg/cmd/roachtest/tests/db_console_cypress.go:99
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runDbConsoleCypressMixedVersions.func1
| pkg/cmd/roachtest/tests/db_console_cypress.go:259
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.runHookStep.Run
| pkg/cmd/roachtest/roachtestutil/mixedversion/steps.go:468
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep.func2
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:268
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.panicAsError
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:819
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:267
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:242
| [...repeated from below...]
Wraps: (4) retryable infrastructure error: could not install [docker]
Wraps: (5) TRANSIENT_ERROR(install_flake)
Wraps: (6) forced error mark
| "error occurred after retries"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (7) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.runWithMaybeRetry
| pkg/roachprod/install/cluster_synced.go:242
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE.func1.1
| pkg/roachprod/install/cluster_synced.go:2580
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (8) error persisted after 3 attempts
Wraps: (9) secondary error attachment
| TRANSIENT_ERROR(apt_problem): exit status 100
| (1) Node 5. Command with error:
| | ``````
| | set -exuo pipefail;
| | # Add Docker's official GPG key:
| | sudo apt-get update;
| | sudo apt-get install -y \
| | ca-certificates \
| | curl \
| | gnupg;
| | sudo install -m 0755 -d /etc/apt/keyrings;
| | curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --no-tty --batch --yes --dearmor -o /etc/apt/keyrings/docker.gpg;
| | sudo chmod a+r /etc/apt/keyrings/docker.gpg;
| |
| | # Add the repository to Apt sources:
| | echo \
| | "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
| | $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
| | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null;
| | sudo apt-get update;
| |
| | # Install
| | sudo apt-get install -y docker-ce;
| | sudo usermod -aG docker ubuntu;
| |
| | # Verify
| | sudo docker run hello-world
| |
| | ``````
| | ... do chmod a+r /etc/apt/keyrings/docker.gpg
| | + sudo tee /etc/apt/sources.list.d/docker.list
| | ++ dpkg --print-architecture
| | ++ . /etc/os-release
| | +++ PRETTY_NAME='Ubuntu 22.04.4 LTS'
| | +++ NAME=Ubuntu
| | +++ VERSION_ID=22.04
| | +++ VERSION='22.04.4 LTS (Jammy Jellyfish)'
| | +++ VERSION_CODENAME=jammy
| | +++ ID=ubuntu
| | +++ ID_LIKE=debian
| | +++ HOME_URL=https://www.ubuntu.com/
| | +++ SUPPORT_URL=https://help.ubuntu.com/
| | +++ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
| | +++ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
| | +++ UBUNTU_CODENAME=jammy
| | ++ echo jammy
| | + echo 'deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable'
| | + sudo apt-get update
| | + sudo apt-get install -y docker-ce
| | E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/jammy/pool/stable/amd64/containerd.io_2.2.1-1%7eubuntu.22.04%7ejammy_amd64.deb 404 Not Found [IP: 3.161.193.42 443]
| | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
| Wraps: (2) TRANSIENT_ERROR(apt_problem)
| Wraps: (3) exit status 100
| Error types: (1) *hintdetail.withDetail (2) errors.TransientError (3) *exec.ExitError
Wraps: (10) secondary error attachment
| TRANSIENT_ERROR(apt_problem): exit status 100
| (1) Node 5. Command with error:
| | ``````
| | set -exuo pipefail;
| | # Add Docker's official GPG key:
| | sudo apt-get update;
| | sudo apt-get install -y \
| | ca-certificates \
| | curl \
| | gnupg;
| | sudo install -m 0755 -d /etc/apt/keyrings;
| | curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --no-tty --batch --yes --dearmor -o /etc/apt/keyrings/docker.gpg;
| | sudo chmod a+r /etc/apt/keyrings/docker.gpg;
| |
| | # Add the repository to Apt sources:
| | echo \
| | "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
| | $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
| | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null;
| | sudo apt-get update;
| |
| | # Install
| | sudo apt-get install -y docker-ce;
| | sudo usermod -aG docker ubuntu;
| |
| | # Verify
| | sudo docker run hello-world
| |
| | ``````
| | ... do chmod a+r /etc/apt/keyrings/docker.gpg
| | + sudo tee /etc/apt/sources.list.d/docker.list
| | ++ dpkg --print-architecture
| | ++ . /etc/os-release
| | +++ PRETTY_NAME='Ubuntu 22.04.4 LTS'
| | +++ NAME=Ubuntu
| | +++ VERSION_ID=22.04
| | +++ VERSION='22.04.4 LTS (Jammy Jellyfish)'
| | +++ VERSION_CODENAME=jammy
| | +++ ID=ubuntu
| | +++ ID_LIKE=debian
| | +++ HOME_URL=https://www.ubuntu.com/
| | +++ SUPPORT_URL=https://help.ubuntu.com/
| | +++ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
| | +++ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
| | +++ UBUNTU_CODENAME=jammy
| | ++ echo jammy
| | + echo 'deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable'
| | + sudo apt-get update
| | + sudo apt-get install -y docker-ce
| | E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/jammy/pool/stable/amd64/containerd.io_2.2.1-1%7eubuntu.22.04%7ejammy_amd64.deb 404 Not Found [IP: 3.161.193.42 443]
| | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
| Wraps: (2) TRANSIENT_ERROR(apt_problem)
| Wraps: (3) exit status 100
| Error types: (1) *hintdetail.withDetail (2) errors.TransientError (3) *exec.ExitError
Wraps: (11) Node 5. Command with error:
| ``````
| set -exuo pipefail;
| # Add Docker's official GPG key:
| sudo apt-get update;
| sudo apt-get install -y \
| ca-certificates \
| curl \
| gnupg;
| sudo install -m 0755 -d /etc/apt/keyrings;
| curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --no-tty --batch --yes --dearmor -o /etc/apt/keyrings/docker.gpg;
| sudo chmod a+r /etc/apt/keyrings/docker.gpg;
|
| # Add the repository to Apt sources:
| echo \
| "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
| $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null;
| sudo apt-get update;
|
| # Install
| sudo apt-get install -y docker-ce;
| sudo usermod -aG docker ubuntu;
|
| # Verify
| sudo docker run hello-world
|
| ``````
| ... do chmod a+r /etc/apt/keyrings/docker.gpg
| + sudo tee /etc/apt/sources.list.d/docker.list
| ++ dpkg --print-architecture
| ++ . /etc/os-release
| +++ PRETTY_NAME='Ubuntu 22.04.4 LTS'
| +++ NAME=Ubuntu
| +++ VERSION_ID=22.04
| +++ VERSION='22.04.4 LTS (Jammy Jellyfish)'
| +++ VERSION_CODENAME=jammy
| +++ ID=ubuntu
| +++ ID_LIKE=debian
| +++ HOME_URL=https://www.ubuntu.com/
| +++ SUPPORT_URL=https://help.ubuntu.com/
| +++ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
| +++ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
| +++ UBUNTU_CODENAME=jammy
| ++ echo jammy
| + echo 'deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable'
| + sudo apt-get update
| + sudo apt-get install -y docker-ce
| E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/jammy/pool/stable/amd64/containerd.io_2.2.1-1%7eubuntu.22.04%7ejammy_amd64.deb 404 Not Found [IP: 3.161.193.32 443]
| E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Wraps: (12) TRANSIENT_ERROR(apt_problem)
Wraps: (13) exit status 100
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) errors.TransientError (6) *markers.withMark (7) *withstack.withStack (8) *errutil.withPrefix (9) *secondary.withSecondaryError (10) *secondary.withSecondaryError (11) *hintdetail.withDetail (12) errors.TransientError (13) *exec.ExitError
Test: db-console/mixed-version-cypress
Messages: failed to install docker
(require.go:1357).NoError: FailNow called
(mixedversion.go:893).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/db-console/mixed-version-cypress/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-20920785-1767338005-35-n5cpu4-0001 | 35.185.109.106 | 10.142.2.213 |
| teamcity-20920785-1767338005-35-n5cpu4-0002 | 34.23.186.201 | 10.142.2.254 |
| teamcity-20920785-1767338005-35-n5cpu4-0003 | 34.74.212.94 | 10.142.2.183 |
| teamcity-20920785-1767338005-35-n5cpu4-0004 | 34.74.53.15 | 10.142.2.216 |
| teamcity-20920785-1767338005-35-n5cpu4-0005 | 34.74.234.160 | 10.142.0.11 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- mvtDeploymentMode=separate-process
- mvtVersions=v25.3.6 → v25.4.2 → v26.1.0-beta.2-dev-697c33f7d0997004dca91195440a97a833258eaf (release-26.1)
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20920785/db-console-mixed-version-cypress/1767349523272/1767351017068)

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*install_flake\): error persisted after 3 attempts: TRANSIENT_ERROR\(apt_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-58330