oxidecomputer / oxidecomputer/propolis

recentish Alpine don't love >64 vCPUs

Open
#1,147 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug guest-os
Dominant language
Rust
Stars
270
Forks
42
Avg merge
4d 5h
Merged PRs (30d)
6

Description

I noticed that a VM on a racklette seemed to get stuck at booting 'Linux virt' when I gave it a bunch (140) vCPUs. if you look at per-processor load, a bunch of vCPUs end up sitting at 100% in guest code, but it never really makes progress towards booting.

I've narrowed this down to a Linux change somewhere between 6.12.31 (Alpine 3.22.0) and 6.12.38 (Alpine 3.22.1) but the changelogs for 6.12.33 to 6.12.36 are ... voluminous.

this is very easily reproducible on any helios system and propolis-standalone, you really do just need to set cpus = 66 or whatever. I was using 80 as a nice round number. I'm probably holding Linux wrong but I couldn't quickly get log lines out to the serial port while it's booting, so I'm not sure where it gets stuck. bit I did notice that all vCPUs get busy for a short period early in boot that seems to line up with smpboot: x86: Booting SMP configuration:, a line about all the CPUs, and then smp: Brought up 1 node, 64 CPUs.

I basically expect that this is https://github.com/oxidecomputer/propolis/issues/940 in some weird way, but it'd be good to be sure what this is or isn't.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue in propolis-standalone with an Alpine virt guest and cpus = 66 or 80, comparing Alpine 3.22.0 (Linux 6.12.31) with 3.22.1 (Linux 6.12.38). Start by checking serial boot output around x86 SMP bring-up and the reported 64-CPU limit, then determine whether this is related to issue 940; done means the regression is explained and the affected behavior is fixed or clearly ruled out.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.