Feasibility: direct-HMC PowerVM provider for M5 LPAR composition

Related: #1760, #1761, #1792, #1782, #1776

## Problem

#1760 scopes M4 to *pre-created* LPARs and lists "creating, deleting, resizing, or migrating
PowerVM LPARs" as a non-goal, on the grounds that "dynamic LPAR creation is not documented MAAS
behavior." An online feasibility study confirms that reasoning and extends it: **neither candidate
M4 control plane can compose an LPAR at all.**

- MAAS's `hmc` power driver (`src/provisioningserver/drivers/power/hmc.py`) is paramiko SSH over
`chsysstate`/`lssyscfg`, declares `can_set_boot_order = False` and `can_probe = False`, hardcodes
`--bootstring network-all` on every power-on, and raises `NotImplementedError` for reset. MAAS
cannot enlist or create LPARs, only power ones that already exist.
- Beaker's PowerVM support is the `lpar` power script delegating to `fence_lpar`, an SSH
screen-scraper offering `on|off|reboot|status` only. It explicitly rejects `interrupt` — which
answers question 3 of #1792 ahead of the live evaluation.

So the roadmap's M5 "LPAR activation + HMC" (`docs/design/top-level-design.md:583`) needs a provider
that drives the HMC directly rather than through a control plane. This issue records what is
documented today for that path, so #1761 can fix the M4/M5 boundary against evidence instead of an
assumption, and so the M5 design starts from a known surface.

Scope studied: LPAR create/delete, processor and memory sizing, dedicated PCIe slot assignment,
power control, and boot order. Network install was deliberately deferred.

## Findings

### Two automation surfaces, and the ecosystem picked one

The HMC exposes an SSH CLI (restricted shell) and a REST API (XML/Atom "UOM" model, async Job
pattern). IBM documents both in the same doc set and positions neither as replacing the other.

**Every production integrator drives the SSH CLI, not REST**: MAAS (paramiko), Beaker
(`fence_lpar`), xCAT (Perl), SUSE openQA (`bootloader_pvm.pm`), and IBM's own `ibm.power_hmc`
Ansible collection for several modules. The only serious REST consumers are IBM's own pypowervm
(a NovaLink-local code drop whose remote-to-HMC auth path the README calls development/test only)
and IBM's Python sample client, untouched since 2015.

Part of the reason: the REST Job layer is largely a wrapper over the CLI. IBM documents job results
as `returnCode` ("the return code value from the invoked command-line interface") and `result`
("contains the STDERR value from the invoked CLI"). REST does not buy a structured error contract —
`HSCLxxxx` message strings get parsed either way.

REST-specific hazards on top of that: session leakage can wedge an HMC until reboot (an IBM support
note records a 5-minute polling script accumulating 3000+ sessions); the REST port is migrating
12443 to 443 and V11 is 443-only; no OpenAPI spec exists; and the UOM XSDs live inside an HMC jar
that normal users cannot read, so generated clients break on upgrade — IBM's own sample client died
exactly that way.

CLI costs are real but bounded: one command per SSH invocation (the HMC is a closed appliance that
refuses `sh -c` and `/` in command names), nested-quote escaping on list attributes, and
delimiter-joined `-F` output rather than structured data.

### Port coverage

| Port | HMC primitive | State |
|---|---|---|
| `Provisioner.provision` | `mksyscfg -r lpar` with proc/mem/`io_slots`, then `chsysstate -o on` | Documented |
| `Provisioner.teardown` | `chsysstate -o shutdown --immed` + `rmsyscfg -r lpar` | Documented; service partitions need `service_lpar_id=none` first, else `HSCL3396` |
| `Booter.boot` | `chsysstate -o on [-b norm]`; poll `lssyscfg -F state`; `lsrefcode` for progress codes | Documented |
| `Controller.power` | `chsysstate` `on` / `osshutdown` (needs RMC) / `shutdown --immed --restart` | Documented |
| `Controller.force_crash` | `chsysstate -o dumprestart` (function 22), or `-o shutdown --restart` without `--immed` | Documented, and not IBM i-only — see the man-page comment below. Whether it yields a vmcore the shared retrieve path reads is open; feeds #1782 |
| `Controller.diagnostic_sysrq` | No HMC-side trigger; SysRq over vterm or in-guest SSH | Follows the remote-libvirt in-guest pattern |
| Console ports | `mkvterm` / `rmvterm` over SSH | Documented; **one vterm session per partition**, no REST equivalent |
| `Connector` (debug) | No hypervisor gdbstub on PowerVM | Advertise `drgn-live` only; capability-descriptor restriction |
| `Retriever` | In-guest kdump/fadump vmcore over SSH, two-phase per ADR-0084 | Follows remote-libvirt |
| `Snapshotter` | No LPAR equivalent | Not supported; the port is already optional via `ProviderSupport.supports_snapshots` |
| Discovery | `lssyscfg -r sys`, `lshwres` slot and SR-IOV enumeration | Documented |

### Dedicated PCIe assignment works, at profile granularity

Verified verbatim from the `chsyscfg` man page: `io_slots` is a comma-separated list of
`slot-DRC-index/[slot-IO-pool-ID]/is-required`, e.g. `21010003/3/1`. DRC indexes come from
`lshwres -r io --rsubtype slot -m -F drc_index,phys_loc,description,lpar_name`. Because it is
a list attribute, `io_slots+=` and `io_slots-=` edit single entries without rewriting the list.

Assignment is per PCIe slot. The only granularity limit found is paired/double-wide adapters, where
both physical slots must belong to the same partition. Slots are `required` (activation fails
without the device; no DLPAR removal) or `desired`.

Dynamic add/remove on a running LPAR (`chhwres -r io -o a|r|m -l `) needs a live RMC connection
and OS-side hot-plug cooperation, and DLPAR changes are lost on the next profile activation unless
`sync_curr_profile=1`. Recommendation for M5: treat PCIe assignment as a provision-time profile
decision with reactivation, which matches how KDIVE provisions a System per profile anyway.

### Boot order: a real API gap, but a one-time prerequisite rather than a per-Run operation

Boot **mode** is settable from the HMC — `boot_mode` in the profile, or `chsysstate -o on -b sms`,
with values `norm`, `sms`, `of`, `dd`, `ds`. Netboot is programmable, either via `lpar_netboot` or
the REST `PowerOn_LogicalPartition` job with `OperationType=netboot`.

Persistent boot **device order** has no HMC API. It lives in partition-firmware NVRAM and is set
from inside the OS (`bootlist` on AIX; GRUB/PReP conventions on Linux) or by expect-scraping SMS/OF
menus over a vterm. IBM's own `lpar_netboot` cannot set boot order either — the `-w set_boot_order`
option exists only in xCAT's vendored Perl reimplementation, not in IBM's command. An open IBM RFE
requests precisely this capability (https://ibm-power-systems.ideas.ibm.com/ideas/PVM-I-51), which
is good evidence it does not exist.

**Reading Beaker's source shows why this does not block a provider.** Beaker netboots PowerVM LPARs
in production and never manipulates boot order at all — a search of the tree for `lpar_netboot`,
`bootlist`, `sms`, `boot_mode`, `bootstring`, and `open firmware` returns one hit, a comment. Its
model instead:

1. The LPAR is set **network-first, disk-second, once, out of band** (the manual SMS step an
operator performs today) and never touched again.
2. Beaker controls only what the network serves. `configure_ppc64`
(`LabController/src/bkr/labcontroller/netboot.py:660`) writes `ppc/grub.cfg-` per system
and symlinks `-grub2` to `../boot/grub2/powerpc-ieee1275/core.elf`. It writes that config
in three locations to cover grub2-version and platform differences, one labelled verbatim
`# hack for power VMs` — the LPAR case.
3. **The fallthrough is a one-word GRUB config.** Alongside every per-system config it writes a
default `grub.cfg` containing `exit` (`netboot.py:209`). With no per-system config, GRUB exits to
Open Firmware, which advances to the next boot-list entry: local disk. The machine netboots every
time and falls through when there is nothing to install.
4. Teardown is triggered from inside the guest: the kickstart `%post` calls the lab controller's
`clear_netboot` endpoint (`proxy.py:648`), which unlinks the per-system config.

So the netboot-versus-local-boot decision moves out of client NVRAM (no API) and into the TFTP
server's filesystem (trivially writable). MAAS reaches the same end by the opposite route, hardcoding
`--bootstring network-all` on every power-on — which is why its driver can declare
`can_set_boot_order = False` and still work.

For KDIVE this reduces the gap to an **operator prerequisite** plus a server-side mechanism, with no
need to manage boot order per Run. Caveat on borrowing it wholesale: Beaker's model assumes a network
install path, deliberately deferred in this study, whereas KDIVE stages kernels in-guest and reboots
(ADR-0082). The transferable pieces are the prerequisite framing and the `exit`-fallthrough trick for
first provision and for recovering a wedged System.

Power10 adds a wrinkle: with Secure Boot enabled the normal Open Firmware prompt is replaced by a
restricted macro set (`BOOT_FROM_DEVICE`, `DISPLAY_BOOT_DEVICES`), and IBM states plainly that
"existing tools will require updates to function in this new environment." `setenv boot-device`
scraping does not survive there.

### Platform boundaries worth recording

- **OPAL bare-metal is gone on Power10 and later**, so PowerVM — and therefore an HMC or NovaLink —
is effectively mandatory on current hardware.
- **eBMC Redfish is not an alternative control path.** It operates at managed-system level. Partition
management flows HMC → VMI (Virtualization Management Interface, in the hypervisor) → PHYP, and
IBM documents that users do not interact with VMI directly. No LPAR create, no per-partition boot
control, no console over Redfish.
- Power11 requires HMC V11, which runs only on 7063-CR2 or vHMC; vHMC is a separate purchase and
cannot manage the server hosting it.
- The HMC CLI and REST API carry no separate license. PowerVC does, and is not needed here.

### Nothing in the ecosystem is adoptable as a dependency

No Go HMC client exists at all. pypowervm is an IBM-internal NovaLink drop. The OpenStack PowerVM
drivers are retired — Nova's removal commit calls the in-tree driver "abandonware." IBM's Python
REST sample is dead since 2015. The one Terraform provider for on-prem PowerVM is closed-source
beta from a single author who states it will not be published or open-sourced. `ibm.power_hmc` is
alive (quarterly releases) but GPL-3.0, hardcodes `bootmode: 'norm'`, cannot modify physical I/O
after creation, and shells out to `sshpass` with `use_unsafe_shell=True`.

Useful as reference implementations, in order: **xCAT's HMC code** (the only OSS tool exposing SMS
boot mode as a first-class action, `rpower sms`), **MAAS `hmc.py`** (minimal clean driver),
**openQA `bootloader_pvm.pm`** (the SMS/OF scraping state machine), and **IBM's Ruby SDK** (the
cleanest REST Job polling loop, if REST is ever added).

### Design constraints for M5

1. **Serialize per HMC.** IBM attributes the practical concurrency ceiling to "the processing
capacity of the HMC and contention for HMC locks." KDIVE's advisory-lock discipline should extend
to a per-HMC scope, not just per-System. Scale ceiling is 48 systems / 2000 partitions per HMC.
2. **Probe the HMC version at discovery.** REST port cutover, V10 vs V11 CLI deltas, Power10 Secure
Boot restrictions, and `chhwres -l` comma-lists (M1060+) all vary by level.
3. **Argument construction is an injection surface.** The `ibm.power_hmc` interpolated-shell pattern
is the anti-pattern; IBM's own tooling shows four levels of backslash escaping on list
attributes. Arguments must be structured and tested, never string-glued.
4. **The vterm is exclusive.** One session per partition, and orphans must be force-closed with
`rmvterm`. The provider must arbitrate console reads against any netboot scraping. Bears on #1776.
5. **Credentials.** The HMC uses key-based auth registered with `mkauthkeys`; `$HOME/.ssh` on the
appliance is not writable. Restricted-shell breakouts have a history (CVE-2021-29707,
CVE-2025-1950), so account posture matters. The design doc already requires that HMC tokens never
appear in requests, state rows, or responses (`docs/design/top-level-design.md:326`).
6. **No published latency figures exist** for partition create or power-on. The REST netboot job's
default timeout is 3600000 ms, which is IBM's implied worst case, and openQA polls activation for
60 s before giving up. Per the "state a limit's full contract" invariant, M5 deadlines must be
measured on hardware rather than inherited from libvirt-scale numbers.
7. **Operator prerequisite: LPAR boot order is network-first, disk-second, set once.** Not settable
from the HMC (see the boot-order section), so it belongs in the M5 environment contract next to
HMC reachability and credentials, the same way the `live_vm` tiers declare their host contract.
A provider that assumes it can set boot order per Run has no documented mechanism to do so; one
that treats it as a precondition matches how both Beaker and MAAS ship working Power support.

## Open questions

Blocking a design:

1. ~~Full man-page synopses for `mksyscfg`, `chhwres`, `chsysstate`, `rmsyscfg`, `lpar_netboot`, and
`lsrefcode`.~~ **Resolved** — all six retrieved and verified; see the man-page comment below.
Two corrections to this body resulted.
2. ~~`lsrefcode` options and output attributes.~~ **Resolved** — it is the boot-progress polling
hook, and it does not require the exclusive vterm.
3. Real activation, create, and power-on latencies, to set the provider's deadline contracts.

Should answer:

4. Documented concurrent **SSH** session limits on an HMC. Only WebUI session limits were found
(`max_webui_sessions_per_user=100`, `max_webui_sessions=1000`).
5. ~~Whether `chhwres` has a `--rmdev` flag.~~ **Resolved** — it does not. `chhwres` issues the VIOS
`rmdev` internally when removing a *virtual* adapter from a VIOS partition; that is the source of
the confusion.
6. ~~Whether `chsysstate -o dumprestart` exists.~~ **Resolved** — it does, and it is not IBM i-only.
This changed the `force_crash` mapping above and raises a new question: does it yield a vmcore
the shared retrieve path can read, or only a platform dump? Tracked against #1782.
7. The HMC version floor for REST JSON support. Current Power10/Power11 docs show JSON request and
response media types, but no release note names the level that introduced it. Only matters if
REST is added later.
8. Whether two PCIe slots sharing one PHB through bifurcation can be assigned to different
partitions. POWER9's per-stack error isolation implies yes; no IBM statement confirms it.
9. `GetNetworkBootDevices_LogicalPartition` job parameters — existence confirmed from the REST API
PDF table of contents only.

## Expected outcome

A recorded feasibility basis for the M5 direct-HMC provider: the surface chosen, the port mapping,
the two gaps (persistent boot order, snapshots) and how each is absorbed, and the open questions
retired or carried into the design. Enough for #1761 to fix the M4/M5 boundary with evidence, and
for an M5 spec to start without re-deriving the surface.

No KDIVE code changes.

## Proposed approach

- Build on the **SSH CLI** as the provider spine. It covers the full required surface including the
console, which REST cannot; it is what every production integrator chose; IBM announces CLI
deprecations a release ahead and keeps old forms working; and it fits the existing remote-provider
idioms (SSH transport, secrets by reference, in-guest install and crash capture).
- Treat REST as an optional later addition if the structured netboot job or quick-property reads
justify a second transport.
- **Do not manage boot order at all.** Make network-first/disk-second a declared operator
prerequisite (constraint 7) and control the boot target server-side, the way Beaker does: serve a
per-System config to netboot, withdraw it to fall through to local disk. This replaces the earlier
proposal to drive the OS-side `bootlist`, which added a mechanism for something two production
systems solve by not needing it. Build no SMS-menu scraping — and note that Power10 Secure Boot
would constrain that fallback anyway.
- Advertise no snapshot support and no `gdbstub` transport in the capability descriptor.
- Verify the open questions against a live HMC before committing syntax to a design.

Start with src/provisioningserver/drivers/power/hmc.py and docs/design/top-level-design.md:583, then compare the cited Beaker netboot.py and related provider references. Verify the documented HMC surfaces and constraints against the stated M5 scope. Done means the feasibility boundary and implementation requirements are recorded clearly enough to guide a later provider design.

Written by the indexing model from the issue text.