cockroachdb / cockroachdb/cockroach
roachprod: bump pinned Azure image with broken MANA on v6-series VMs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
roachprod pins the Azure Ubuntu image to `22.04.202312060` (Dec 2023, kernel `6.2.0-1018-azure`) in [azure.go#L1080](https://github.com/cockroachdb/cockroach/blob/00a3c2c5b3f89a6d8c9dfd6981ed85ac09944816/pkg/roachprod/vm/azure/azure.go#L1080). On v6-series VMs (e.g. `Standard_D4s_v6`), which have MANA NICs only (no Mellanox), this kernel's `mana` driver fails to probe:
```
mana 7870:00:00.0: HWC: Failed hw_channel req: 0xc00000bb
mana 7870:00:00.0: gdma probe failed: err = -71
```
All traffic then takes the unaccelerated synthetic `hv_netvsc` path. Under load (e.g. IMPORT during test setup), inter-node RPC latency spikes to seconds (`latency jump (prev avg 0.45ms, current 4472.26ms)`), gRPC connections flap, and DistSQL flows die. The flake is nondeterministic (host-placement-dependent) but the degraded networking is deterministic: every v6 VM runs unaccelerated. Dv5/Ebsv5-series are also [eligible to land on MANA hosts](https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-mana-existing-sizes) now, so exposure will grow.
**Verified fix**
Booting a `Standard_D4s_v6` in eastus with the current jammy gen2 image `22.04.202607140` (kernel `6.8.0-1062-azure`), MANA probes cleanly and traffic flows through the VF:
```
mana 7870:00:00.0 eth1: joined to eth0
hv_netvsc ... eth0: Data path switched to VF: enP30832s1
vf_rx_packets: 9439 / vf_tx_dropped: 0
```
**Next steps**
- [ ] Confirm a version string available for all three SKUs used by [`imageSKU`](https://github.com/cockroachdb/cockroach/blob/00a3c2c5b3f89a6d8c9dfd6981ed85ac09944816/pkg/roachprod/vm/azure/azure.go#L1033-L1045) (`22_04-lts`, `22_04-lts-gen2`, `22_04-lts-arm64`)
- [ ] Bump the pin; watch Azure nightlies (fleet-wide kernel jump 6.2 -> 6.8)
Jira issue: CRDB-66505
Contributor guide
Research direction
Inspect pkg/roachprod/vm/azure/azure.go around imageSKU at lines 1033-1045 and the pinned image at line 1080. Check that the current jammy Gen2 image version is available for 22_04-lts, 22_04-lts-gen2, and 22_04-lts-arm64, then update the pin and verify the selected image supports working MANA networking on the affected VM series.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, ubuntu
- Domain
- cloud, infrastructure, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100