bytecodealliance / bytecodealliance/wasmtime

Is differential fuzzing the spec interpreter still worth it?

Open
#13,561 2 comments 0 reactions 0 assignees View on GitHub
fuzzing
Dominant language
Rust
Stars
18.6k
Forks
1.8k
Avg merge
1d 19h
Merged PRs (30d)
121

Description

An oss-fuzz bug just [got opened](https://oss-fuzz.com/testcase-detail/5044447208603648) which is a timeout in the spec interpreter. In investigating it locally I've discoverd that when my gcc updated a few days ago I can no longer compile the C code in `ocaml-boxroot-sys`, which `ocaml-interop`, what we use in Rust, depends on. The latest version of `ocaml-interop` does at least compile its C code for me, but the new Rust APIs are, at my read, fundamentally incompatible with how we want to invoke OCaml. There's no way that I can see which would initialize the OCaml runtime and a way that can be reused without having to thread it across fuzz executions.

This led me to a broader question of possibly removing the differential fuzzing against the spec interpreter. The points leading me to this conclusion are:

* We haven't had any differential fuzz issues found in years now at this point.
* In the meantime wasmi is on-parity with the spec interpreter in terms of wasm features.
* Differentially fuzzing against N engines, I believe, is more-or-less the same as differential fuzzing against N+1 engines. We'd still be differential-fuzzing against V8, wasmi, and Cranelift vs Winch.
* The spec interpreter integration hasn't been touched in many years. This is still Conrad's old fork of the spec interpreter which fixed some original performance issues, but we haven't updated it in quite awhile. The Rust integration itself is quite old.

Basically it's currently got a fuzz bug, I can't investigate locally due to our Rust deps using now-broken C code, and I also fear we've been keeping it running for quite some time without a whole lot of benefit.

I wanted to canvas others, however. How do others feel about this? (cc @fitzgen and @cfallin)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.