cargo xtask download selects CockroachDB and ClickHouse by OS only, silently fetching amd64 binaries on any non-amd64 host

Description

The README states that simulated Omicron "can be executed on Linux, Mac, and
illumos". That claim is not qualified by architecture, but the prerequisite
downloader is: it picks CockroachDB and ClickHouse builds by operating system
with the architecture hardcoded.

dev-tools/downloader/src/lib.rs, download_cockroach() (line 690):

let build = match os {
    Os::Illumos => "illumos-amd64",
    Os::Linux => "linux-amd64",
    Os::Mac => "darwin-amd64",
};

There is no arch() call in that function at all. The same pattern applies to
ClickHouse, and the checksum files carry a single value per OS
(tools/cockroachdb_checksums, tools/clickhouse_checksums each hold only
CIDL_SHA256_DARWIN / CIDL_SHA256_LINUX / CIDL_SHA256_ILLUMOS).

The file does know about architecture — Arch::X86_64 and Arch::Aarch64
exist, arch() is defined, and download_cargo_hack() uses them properly. Only
the two database downloads ignore it.

The consequence is that on aarch64 Linux the download step succeeds — the
amd64 tarball matches the amd64 checksum — and then fails at the
"Checking that binary works" step with an error that names no cause:

Error: Failed to download Clickhouse

Caused by:
    0: Failed to run out/clickhouse/clickhouse
    1: Exec format error (os error 8)

"Exec format error" is the only hint that an architecture mismatch happened, and
it appears under a "Failed to download" heading, which is misleading — the
download was fine.

A related inconsistency in the same area: download_cargo_hack() builds an
arch-aware URL (cargo-hack-aarch64-unknown-linux-gnu.tar.gz) but validates it
against tools/cargo_hack_checksum, which has one CIDL_SHA256_LINUX entry.
On aarch64 the correct tarball is downloaded and then always fails checksum
verification. So of the three, one ignores arch in the URL and one ignores it in
the checksum; they fail in opposite directions.

Steps to reproduce

1. On aarch64 Linux, clone omicron (full clone — see the shallow-clone issue).
2. ./tools/install_builder_prerequisites.sh -y

Expected result

Either a native aarch64 build is fetched, or the tool fails immediately with a
clear message — e.g. "CockroachDB is only published for x86_64 on Linux;
aarch64 is unsupported" — before downloading anything.

Actual result

An amd64 binary is installed into out/, verified against its (correct) amd64
checksum, and then fails to execute. cargo-hack fails checksum verification
instead. Neither error names the architecture as the cause.

Suggested fix

Two options, either acceptable:

1. Fail fast and clearly. Match on (os, arch) as
   download_cargo_hack() already does, and bail! with an explicit
   "unsupported architecture" message for combinations with no published build.
   This is a small change and would have turned a confusing runtime failure into
   an immediate, accurate one.

2. Support the architecture. Upstream publishes linux-arm64 CockroachDB
   from v22.2 onward and aarch64 ClickHouse for the pinned 23.8.7.24. Note this
   is not currently possible for the pinned CockroachDB version: no arm64
   build of v22.1 exists, so honouring tools/cockroachdb_version on aarch64
   is not achievable without a version bump.

Separately, the cargo_hack_checksum file should carry per-architecture entries
to match its arch-aware URL, or the URL should be pinned to one architecture.

Workaround

Install native aarch64 CockroachDB and ClickHouse into out/cockroachdb/bin/
and out/clickhouse/ after running install_builder_prerequisites.sh —
running it afterwards re-downloads and overwrites them.

Environment

omicron df990b0578fbee4afcf805423a20d85e23544e0d (2026-09-04)
platform aarch64-unknown-linux-gnu (Ampere Altra Q80-30), Ubuntu 24.04
rust 1.98.1 (pinned by rust-toolchain.toml)
mode simulated (cargo xtask omicron-dev run-all)

---

Disclosure: this issue was investigated and written up with AI assistance
(Claude). Everything in it was measured rather than inferred — the timings,
error output, version numbers and reproduction steps are all from real runs on
real hardware, and where a fix is suggested it is one I am actually running. I
have read it through before filing. Happy to clarify anything or test a patch.

Start in dev-tools/downloader/src/lib.rs with download_cockroach(), the ClickHouse download logic, and download_cargo_hack(); compare their architecture handling with arch(). Then inspect tools/cockroachdb_checksums, tools/clickhouse_checksums, and cargo_hack_checksum. Done means supported architectures use matching artifacts and checksums, while unsupported combinations fail clearly before execution; verify with the prerequisite installer on aarch64 Linux.