bytecodealliance / bytecodealliance/wasmtime

Spectre mitigations: add a mode that monitors branch mispredictions and dynamically turns on fences

Open
#8,175 6 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
18.6k
Forks
1.8k
Avg merge
1d 18h
Merged PRs (30d)
126

Description

In discussion today with @fitzgen, @jameysharp, @elliottt and @lpereira, we were considering the idea to [dynamically monitor branch mispredictions](https://arxiv.org/abs/2110.04751) and isolate execution of any Wasm instance that had used up a "misspeculation quota". I realized that actually what we could do is (effectively) turn off speculation -- you run out, you can't use it anymore! -- by dynamically inserting `lfence`s.

Specifically: the (one?) neat thing about fully coherent icaches on x86 is that we can switch out the code that's running, on the fly, even if other threads are in the middle of functions we're switching out, *as long as we're very careful to do it atomically* (state between any two stores is valid code).

Consider the case where we want an `lfence` before every indirect branch (say; or before every branch; orthogonal detail) and we have:

```
...
mov rax, ... # compute branch target (e.g. from br_table)
nop # space for `lfence` (3 bytes)
nop
nop
jmp rax
```

we can replace the three bytes of `nop` (`0x90, 0x90, 0x90`) with `lfence` (`0x0f`, `0xae`, `0xe8`) if we want to "turn off speculation" for this module for a little bit.

There are at least three ways to do that on an x86 machine (with coherent icaches):
- Do an atomic store to code memory. For this we'd need W+X mappings temporarily, and an extra `nop` to make this a 32-bit region we could overwrite with one 32-bit store.
- Above, but switch from R+X to R+W; take the SIGBUS from any running thread, temporarily hold, and release when we switch the mapping back (via a futex?).
- The one I like best: keep another version of the code segment around, and mmap it over the first.

The last one is pretty neat: `mmap` is atomic with respect to every other thread (appears as a single store in the total store order; it must, because if other thread had it mapped, it would receive an IPI, which is a synchronizing edge). So we basically "yank out the code ROM and replace it" in between instructions, and the new code doesn't speculate.

Using this, we can build a control loop in a separate thread that monitors mispredict counters, and can flip the switch at will for any module that has excessive counts. It doesn't have to be a one-way trapdoor: a module could have a "mispredict quota" per time unit, and could reset to the fast code (no `lfence`s) after a set period. There is no impact on other modules -- it only impacts the module with the mispredicts.

Finally, I suspect this will be a bit harder on non-coherent-icache architectures (aarch64, riscv64), but actually maybe the "mmap a new thing on top of running code" is enough of a jolt to yoink all other cores into coherent happiness again. Note that I haven't tested that!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.