Dstack-TEE / Dstack-TEE/dstack

Normalize the Linux setup header so RTMR[1] stops depending on the QEMU version

Open
#1,185 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
544
Forks
96
Avg merge
17h 57m
Merged PRs (30d)
117

Description

## Summary

Make RTMR[1] independent of the QEMU version by normalizing the Linux setup
header, so `dstack-mr` can go back to a single kernel digest.

Follow-up to #1183, which fixes the immediate breakage by recording **two**
kernel Authenticode digests and picking one from the host-declared
`vm_config.qemu_version`. That works, but it leaves the verifier carrying a
QEMU-version branch forever. This issue removes the branch instead.

## Background

QEMU rewrites part of the Linux setup header before serving the kernel over
fw_cfg, and OVMF measures the result into RTMR[1]. QEMU commit `a7542a38f399`
("x86/loader: Don't update kernel header for CoCo VMs", first in 10.2.0) stopped
doing that for confidential guests, so the same image measures differently on
QEMU <= 10.1 vs >= 10.2. See #1183 for the full analysis.

Measured empirically on a real dstack image, QEMU's rewrite touches **4 regions,
11 bytes**:

```
0x0210..0x0212 orig=[00,01] patched=[b0,81] type_of_loader, loadflags
0x0219..0x021f orig=[00 x6] patched=[10,54,7f,c0,62,a9] ramdisk_image, ramdisk_size
0x0224..0x0226 orig=[a0,50] patched=[00,fe] heap_end_ptr
0x022a..0x022b orig=[00] patched=[02] cmd_line_ptr
```

## Proposal

Normalize the setup header on both sides so the measured bytes are
version-independent **and** still equal the shipped file byte for byte:

1. **Image build** — normalize the shipped `bzImage` in place.
2. **OVMF** — apply the same normalization to the `kernel` blob before it is
measured and loaded.
3. **dstack-mr** — drop `patched_kernel_authenticode`, `patch_kernel()`, the
version branch, the `qemu_patches_kernel_header` escape hatch, and the
`memory_size == 2 GiB || >= 0xB0000000` restriction (that guard only exists
because the patched digest depends on guest RAM).

Normalization is idempotent, so it produces the same bytes whether or not QEMU
rewrote anything. Because the shipped file is normalized too, the measured value
remains a plain Authenticode hash of the artifact in `sha256sum.txt` — no
dstack-specific derived digest.

### Normalization rule

Defined against the boot protocol, **not** against QEMU's behavior, so it does
not go stale when QEMU changes:

> Zero every field `Documentation/arch/x86/boot.rst` types as **`write`**
> (bootloader writes it; the kernel supplies no value), and clear
> `CAN_USE_HEAP` (0x80) in `loadflags`. Leave every `modify` field alone.

That covers `type_of_loader` (0x210), `ramdisk_image`/`ramdisk_size`
(0x218/0x21c), `heap_end_ptr` (0x224), `ext_loader_ver`/`ext_loader_type`
(0x226/0x227), `cmd_line_ptr` (0x228), `hardware_subarch[_data]` (0x23c/0x240)
and `setup_data` (0x250).

**Do not** widen this to "all bootloader-writable fields". `modify` fields carry
real kernel-supplied values and zeroing them breaks the kernel:

| field | type | offset | build-time value |
| --- | --- | --- | --- |
| `code32_start` | modify (optional, reloc) | 0x214 | **0x100000** (protected-mode entry) |
| `setup_move_size` | modify (obligatory) | 0x212 | 0x8000 |
| `vid_mode` | modify (obligatory) | 0x1fa | 0xffff |
| `loadflags` | modify (obligatory) | 0x211 | 0x01 |
| `root_flags` | modify (optional) | 0x1f2 | 0x01 |

`vid_mode` is the one `modify` field QEMU can write, but only when the cmdline
contains `vga=` — and the cmdline is measured into RTMR[2], so it cannot be
smuggled in. Leaving it alone is safe.

### Impact on the shipped kernel: 2 bytes

Every `write` field is already zero in a freshly built bzImage **except**
`heap_end_ptr` (0x224 = `0x50a0`). So normalization changes exactly two bytes.

This is safe for non-TDX boot:

- `boot.rst` types it `write (obligatory)`: the bootloader writes it, the kernel
supplies no value.
- `init_heap()` reads it only when the bootloader has set `CAN_USE_HEAP`, and
that flag is itself bootloader-set (zero at build time). Any bootloader that
sets the flag is obliged by the spec to set `heap_end_ptr` too.
- On the EFI-stub path the real-mode setup code never runs at all.
- The PE headers occupy `0x40..0x170`; every setup-header field is outside them,
so the PE structure and EFI-stub entry are untouched.

Secure Boot is not planned, so there is no signature to invalidate and no
sign-vs-normalize ordering constraint.

## Implementation

### 1. Image build (shared by Yocto and mkosi)

Both backends already funnel through `os/image/assemble.sh`
(`os/yocto/mkimage.sh` execs it; mkosi calls it from `mkosi.postoutput`), so one
insertion point covers both. Add a helper next to the existing
`authenticode_hash.py` / `kernel-cmdline.sh`, e.g.
`os/image/normalize-kernel-header.py`, and call it right after the kernel is
staged:

```diff
verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage"
+ verbose "$(dirname "${BASH_SOURCE[0]}")/normalize-kernel-header.py" "${OUTPUT_DIR}/bzImage"
```

`assemble.sh:413` runs before `tdx-measurement-cbor` (`:469`) and before
`CHECKSUM_FILES` (`:590`), so the measurement document and `sha256sum.txt`
automatically cover the normalized kernel.

The script should be idempotent and should fail loudly if the setup header does
not look like a supported boot protocol version.

**Open question:** the optional UKI image embeds the kernel in a `.linux` PE
section. Check whether it needs the same treatment or is unaffected because it
is not loaded via `-kernel`.

### 2. OVMF

`OvmfPkg/QemuKernelLoaderFsDxe` builds the virtual `kernel` file from
`QemuFwCfgItemKernelSetupData ++ QemuFwCfgItemKernelData` (see
`mKernelBlobItems`). That blob is what `gBS->LoadImage` measures **and** runs on
the EFI-stub path, so normalizing it there keeps measured bytes == loaded bytes
== file bytes — no "attested something other than what ran" gap.

Note `X86QemuLoadImageLib` keeps a separate `SetupBuf` for the legacy
`LoadLinux*` path; dstack does not use it and it should not be touched.

Worth attempting upstream: this is the missing half of Gerd Hoffmann's 2024
series (`f2594d92844`, "x86/loader: expose unpatched kernel", whose commit
message ends "Needs OVMF changes too to be actually useful") and it fixes the
same class of Secure Boot breakage that motivated that series. Note that the
`etc/boot/kernel` fw_cfg file it added only exists in QEMU >= 10.1, which is why
normalization is preferable for us: it works on 8.2.2 too.

### 3. dstack-mr / dstack-types

Only after images with the new OVMF are the floor. Until then #1183's
`patch_kernel_header` switch is still needed to verify CVMs running older
firmware.

## Documentation

- `os/image/README.md` — what the script does and why, so a bit-for-bit
reproducibility check does not trip over `bzImage` differing from the raw
kernel build output.
- `docs/security/security-model.md` — replace the "A host-declared QEMU version
selects between digests" section: once this lands the host no longer has that
choice at all, which is a strictly stronger property.
- `os/yocto/repro-build/check.sh` — make sure the normalization step is part of
the reproducibility story.

## Ordering

Land after #1183. That PR is the fix for the currently broken production path
and it provides the `patch_kernel_header` switch this transition depends on.

Contributor guide

Open the contributing guide

Research direction

Read os/image/assemble.sh around the kernel staging, tdx-measurement-cbor, and CHECKSUM_FILES steps, then inspect OvmfPkg/QemuKernelLoaderFsDxe and the existing dstack-mr kernel-digest handling. Check os/image/README.md, docs/security/security-model.md, and os/yocto/repro-build/check.sh for required documentation and reproducibility updates. Done means both image and OVMF paths normalize consistently, with the QEMU-version digest branch removable after the firmware transition.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, shell
Domain
build-system, operating-systems, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.