cockroachdb / cockroachdb/cockroach

roachprod: bump pinned Azure image with broken MANA on v6-series VMs

Open
#173,178 2 comments 0 reactions 0 assignees View on GitHub
A-roachprod C-bug O-agent T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

roachprod pins the Azure Ubuntu image to `22.04.202312060` (Dec 2023, kernel `6.2.0-1018-azure`) in [azure.go#L1080](https://github.com/cockroachdb/cockroach/blob/00a3c2c5b3f89a6d8c9dfd6981ed85ac09944816/pkg/roachprod/vm/azure/azure.go#L1080). On v6-series VMs (e.g. `Standard_D4s_v6`), which have MANA NICs only (no Mellanox), this kernel's `mana` driver fails to probe:

```
mana 7870:00:00.0: HWC: Failed hw_channel req: 0xc00000bb
mana 7870:00:00.0: gdma probe failed: err = -71
```

All traffic then takes the unaccelerated synthetic `hv_netvsc` path. Under load (e.g. IMPORT during test setup), inter-node RPC latency spikes to seconds (`latency jump (prev avg 0.45ms, current 4472.26ms)`), gRPC connections flap, and DistSQL flows die. The flake is nondeterministic (host-placement-dependent) but the degraded networking is deterministic: every v6 VM runs unaccelerated. Dv5/Ebsv5-series are also [eligible to land on MANA hosts](https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-mana-existing-sizes) now, so exposure will grow.

**Verified fix**

Booting a `Standard_D4s_v6` in eastus with the current jammy gen2 image `22.04.202607140` (kernel `6.8.0-1062-azure`), MANA probes cleanly and traffic flows through the VF:

```
mana 7870:00:00.0 eth1: joined to eth0
hv_netvsc ... eth0: Data path switched to VF: enP30832s1
vf_rx_packets: 9439 / vf_tx_dropped: 0
```

**Next steps**

- [ ] Confirm a version string available for all three SKUs used by [`imageSKU`](https://github.com/cockroachdb/cockroach/blob/00a3c2c5b3f89a6d8c9dfd6981ed85ac09944816/pkg/roachprod/vm/azure/azure.go#L1033-L1045) (`22_04-lts`, `22_04-lts-gen2`, `22_04-lts-arm64`)
- [ ] Bump the pin; watch Azure nightlies (fleet-wide kernel jump 6.2 -> 6.8)

Jira issue: CRDB-66505

Contributor guide

Open the contributing guide

Research direction

Inspect pkg/roachprod/vm/azure/azure.go around imageSKU at lines 1033-1045 and the pinned image at line 1080. Check that the current jammy Gen2 image version is available for 22_04-lts, 22_04-lts-gen2, and 22_04-lts-arm64, then update the pin and verify the selected image supports working MANA networking on the affected VM series.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, ubuntu
Domain
cloud, infrastructure, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.