Running kAFL on a mainline kernel: status and design

kAFL now runs with hardware Intel PT coverage on a stock, unpatched Linux kernel: no `kafl.linux`, no custom KVM module, no out-of-tree PT driver. Coverage comes from mainline `perf`'s `intel_pt` PMU in system mode, hypercalls ride the stock KVM VMware I/O backdoor, and snapshot restore uses the upstream dirty ring.

CR3 filtering is the one thing that still wants a kernel change, and it is a 52 line patch to `arch/x86/events/intel/pt.c` rather than a fork. Without it you keep IP-range filtering and host/guest separation, which is enough for a single purpose guest.

This issue documents what the custom kernel actually contains and what replaces each piece.

## What the custom kernel contains

Exact diffstat, vanilla Linux `v6.8` against `IntelLabs/kafl.linux:kvm-nyx-6.8`, scoped to KVM and uapi:

```
arch/x86/include/asm/kvm_host.h | 16 +
arch/x86/kvm/Kconfig | 6 +
arch/x86/kvm/Makefile | 6 +-
arch/x86/kvm/mmu.h | 10 +-
arch/x86/kvm/vmx/nested.c | 134 +++++
arch/x86/kvm/vmx/nested.h | 6 +
arch/x86/kvm/vmx/vmx.c | 223 +++++++-
arch/x86/kvm/vmx/vmx.h | 12 +
arch/x86/kvm/vmx/vmx_pt.c | 1168 +++++++++++++++++++++++++++++++++++++++
arch/x86/kvm/vmx/vmx_pt.h | 31 ++
arch/x86/kvm/x86.c | 148 ++++-
include/uapi/linux/kvm.h | 118 ++++
12 files changed, 1856 insertions(+), 22 deletions(-)
```

All of it sits behind a single `CONFIG_KVM_NYX` option (`depends on KVM && KVM_INTEL`), so a kernel built without it is functionally vanilla KVM. `vmx_pt.c` is 63% of the diff. Everything else is plumbing to reach it.

What the uAPI provides:

| Feature | uAPI |
|---|---|
| Guest execution tracing via Intel PT | `KVM_VMX_PT_SETUP_FD`, `ENABLE`, `DISABLE` |
| IP-range filtering, 4 ranges | `CONFIGURE_ADDR0..3`, `ENABLE_ADDR*`, `DISABLE_ADDR*` |
| CR3 filtering, single and multi context | `CONFIGURE_CR3`, `ENABLE_CR3`, `CONFIGURE_MULTI_CR3` |
| Trace buffer management | `GET_TOPA_SIZE`, `CHECK_TOPA_OVERFLOW` |
| Page dump breakpoints and MTF single stepping | `SET_PAGE_DUMP_CR3`, `ENABLE_MTF` |
| Fast dirty page logging | `KVM_VMX_FDL_*` |
| VM exit reasons for every hypercall | `KVM_EXIT_KAFL_*` (100 to 134) |

The FDL row is the useful precedent here. `vmx_fdl.c` is already gone: kAFL uses upstream KVM's `KVM_CAP_DIRTY_LOG_RING` (stable since 5.11) for fast snapshot restore, and `KVM_VMX_FDL_SETUP_FD` now returns `-EPERM` as dead code. One of the three custom drivers was retired in favour of an upstream mechanism without losing anything. The work below does the same for the other one that matters.

## Hypercalls: the VMware I/O backdoor

`enable_vmware_backdoor` is a standard upstream KVM module parameter, part of plain `CONFIG_KVM`. It is not a VMware emulator: it tells KVM to intercept the `#GP` the CPU raises when ring 3 code executes `OUT`/`IN` without I/O permission, and emulate the instruction anyway. The guest needs no `iopl()` or `ioperm()` setup, because the guest kernel's own protection is what triggers the trap KVM catches.

```
sudo modprobe -r kvm_intel kvm
sudo modprobe kvm enable_vmware_backdoor=y
sudo modprobe kvm_intel
```

QEMU-Nyx has supported this for years. `kvm_init()` probes `KVM_CAP_NYX_PT`, falls back when it is absent, checks `/sys/module/kvm/parameters/enable_vmware_backdoor`, and sets `nyx_no_pt_mode`. `handle_vmware_hypercall()` then forwards to `handle_kafl_hypercall()` with `EBX + 100`, so every hypercall behaves identically regardless of whether it arrived by `vmcall` or by the I/O port. The guest selects its transport from the CPUID model string, `NYX vCPU (PT)` against `NYX vCPU (NO-PT)`, set by `-cpu kAFL64-Hypervisor-v2`.

The gap was on the guest side. `libnyx_agent`'s dispatcher detected `nyx_cpu_v2` correctly and then did nothing with it:

```c
case nyx_cpu_v2:
case nyx_cpu_none:
debug_printf("\t# vmcall(0x%x,0x%lx) skipped..\n", id, arg);
return 0;
```

That is now implemented (`outl` to port `0x5658`, magic `0x8080801f`, `EBX = id - 100`, `ECX = arg`), along with the same path in `fs_fuzzer` and the Linux agent. See also #208, where the QEMU side was already described as production ready.

## PT tracing: mainline perf in system mode

The received wisdom is that mainline `perf` cannot trace a KVM guest, so `vmx_pt.c` is unavoidable. That is not correct on the hardware kAFL targets.

**VMXON does not stop the trace.** `intel_pt_handle_vmx()` opens with:

```c
/* PT plays nice with VMX, do nothing */
if (pt_pmu.vmx)
return;
```

`pt_pmu.vmx` comes from `IA32_VMX_MISC` bit 14, "tracing post-VMXON" (SDM Vol 3). On a CPU with that bit, VMXON leaves `RTIT_CTL.TraceEn` alone, so there is nothing to save and restore and the function returns immediately. The save/restore logic below it is dead code on those parts. Skylake and later set the bit, which is every kAFL capable host.

**KVM in system mode gets out of the way.** With `pt_mode=0`, `pt_guest_enter()` and `pt_guest_exit()` both return before any `pt_save_msr`/`pt_load_msr`, so a host trace in progress simply continues across VM entry. KVM also actively clears `VM_ENTRY_LOAD_IA32_RTIT_CTL`, `VM_EXIT_CLEAR_IA32_RTIT_CTL`, `VM_ENTRY_PT_CONCEAL_PIP`, `VM_EXIT_PT_CONCEAL_PIP`, `SECONDARY_EXEC_PT_USE_GPA` and `SECONDARY_EXEC_PT_CONCEAL_VMX`. Declining to conceal PIP is exactly what makes the guest half of the trace decodable. KVM's own `capabilities.h` documents the mode:

```
* Processor Trace can operate in one of three modes:
* a. system-wide: trace both host/guest and output to host buffer
* ...
* KVM currently only supports (a) and (c).
```

**CVE-2024-53135 did not close this door.** Commit `aa0d42cacf09` ("KVM: VMX: Bury Intel PT virtualization (guest/host mode) behind CONFIG_BROKEN") is three insertions and one deletion. It gates the `pt_mode` module parameter, so `pt_mode=1` can no longer be selected. `pt_mode` keeps its `PT_MODE_SYSTEM` initializer and every `vmx_pt_mode_is_system()` path is untouched. The bugs in that CVE are all on the guest owned buffer side, which system mode never uses.

**Scoping the trace.** Three mechanisms replace what `KVM_VMX_PT_*` did:

* the PIP `NR` flag separates VMX root from non-root, so libxdc can discard host execution and keep the guest
* IP-range `PERF_EVENT_IOC_SET_FILTER` applies to guest linear addresses
* CR3 filtering needs the kernel patch described below, and is optional

Two problems were not obvious in advance and are worth recording, because anyone reimplementing this will hit them.

**Re-enabled regions never start at a PSB.** Mainline `pt.c` clears `PacketByteCnt` only on the first start, so SDM 36.2.8.3's PSB+ condition never holds again. libxdc searches for a PSB to synchronise on and silently discards everything before the first one it finds, which yields an empty bitmap with no error. The fix is an 18 byte synthetic PSB+PSBEND header prepended to each region before decode, in `nyx/pt_perf.c`.

**The AUX buffer needs no hard reset.** `PERF_EVENT_IOC_RESET` does not touch the AUX ring, and disable/enable does not rewind `aux_head`. Reopening the fd every iteration would be far too slow. It is also unnecessary: tracking `aux_head` and `aux_tail` deltas per iteration costs about 1.08 microseconds and gives the same result.

None of this is novel. `libafl_qemu`'s systemmode module has traced KVM guests through mainline perf since April 2025 using the same `pt_mode=0` mechanism, the same `aux_tail` reset, the same `IA32_VMX_MISC` bit 14 check and the same PIP based host/guest demux.

## Where the code went

| Component | Change | Reference |
|---|---|---|
| Kernel | 52 lines, optional | [`Wenzel/kafl.linux@perf-pt-cr3-filter`](https://github.com/Wenzel/kafl.linux/tree/perf-pt-cr3-filter) (`f9d5da5a`) |
| QEMU-Nyx | ~1876 lines | [`Wenzel/kafl.qemu@pt-perf-backend`](https://github.com/Wenzel/kafl.qemu/tree/pt-perf-backend) (`9b7cefef99`) |
| libxdc | 228 lines of library, 760 of tests | [`Wenzel/kafl.libxdc@nr-tracking`](https://github.com/Wenzel/kafl.libxdc/tree/nr-tracking) (`e87dc1ff`) |
| Guest agents | backdoor transport | [`Wenzel/kafl.targets@vmware-backdoor-hypercall`](https://github.com/Wenzel/kafl.targets/tree/vmware-backdoor-hypercall) (`c41b3d5e`) |
| Fuzzer | `self_check` fix | [`Wenzel/kafl.fuzzer@vmware-backdoor-perf-selfcheck`](https://github.com/Wenzel/kafl.fuzzer/tree/vmware-backdoor-perf-selfcheck) (`a8d5d21`) |

The kernel patch is `perf/x86/intel/pt: allow filtering by CR3 via attr.config2`: 50 added lines in `arch/x86/events/intel/pt.c` and 2 in `pt.h`. It writes `IA32_RTIT_CR3_MATCH` from `perf_event_attr.config2` and rejects reserved bit values with `-EINVAL`. `attr.config2` was chosen over `PERF_EVENT_IOC_SET_FILTER` because the filter ioctl's string grammar is address-range specific and a CR3 value is not an address range. Measured on `6.19.0-cr3`, a non matching CR3 suppresses 99.994% of trace bytes in hardware. It is written against `v6.19` so it is submittable as is; `nyx-6.8-cr3-filter` carries the same change on 6.8 for A/B testing against `vmx_pt`.

On the QEMU side, `nyx/pt.c` was split behind a backend interface (`nyx/pt_backend.h`, 163 lines) with two implementations: `nyx/pt_vmx.c` (194 lines, the existing ioctl consumer, unchanged in behaviour) and `nyx/pt_perf.c` (937 lines, the new one). Select at runtime:

```
-global nyx.pt_backend=perf
```

`vmx_pt` remains the default when the Nyx module is present. Nothing is removed.

libxdc's change is `libxdc_enable_nr_filter()`, which keys on the PIP `NR` bit to keep non-root execution and discard VMX root. The backend advertises whether it needs this (`traces_vmx_root`): a host wide perf event sees both halves in one byte stream, while `vmx_pt` restricts tracing to non-root in hardware through the VMX MSR autoload list and emits no PIP to key on, so enabling the filter on it would suppress everything.

## Reproducing it

Host requirements are three:

1. an Intel CPU with Intel PT, Skylake or newer
2. access to `/dev/kvm`
3. `enable_vmware_backdoor=y` on the stock `kvm` module

On a hybrid part, pin QEMU to P-cores. Intel PT on an E-core will not give you a usable trace.

Build QEMU-Nyx from `pt-perf-backend` against libxdc `nr-tracking`, use a guest agent from `vmware-backdoor-hypercall`, and pass `-global nyx.pt_backend=perf`. `perf_event_paranoid` must permit the event, or the process needs `CAP_PERFMON`.

For CR3 filtering, add `Wenzel/kafl.linux@perf-pt-cr3-filter` on top of `v6.19`. Everything else runs on a distribution kernel as shipped.

Start with the component table and compare the referenced branches, then read arch/x86/events/intel/pt.c, nyx/pt.c, nyx/pt_backend.h, and nyx/pt_perf.c. Reproduce the documented setup only on the stated Intel PT hardware, and treat the work as done when the custom-kernel features, stock-kernel replacements, and remaining CR3-filter requirement are documented accurately.

Written by the indexing model from the issue text.