bytecodealliance / bytecodealliance/wasmtime

<Performance> fuzzbug: Repeated small bounded bulk-memory operations are much slower in Wasmtime than in Wasmer Cranelift

Open
#13,272 4 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Rust
Stars
18.6k
Forks
1.8k
Avg merge
1d 19h
Merged PRs (30d)
121

Description

## Describe the bug

Repeated small bounded bulk-memory operations appear to be much slower in Wasmtime than in Wasmer Cranelift in a minimal microbenchmark family.

I first found this in generated differential tests for `memory.copy`, then reduced and checked it with a smaller reproducer plus several controls. The slowdown is not limited to one seed, and it is still present after varying copy length and src/dst relation.

[test_cases.zip](https://github.com/user-attachments/files/27391229/test_cases.zip)

The clearest primary reproducer I found is:
- `primary_reproducer_memory_copy_len32.wat`

Useful supporting controls are:
- `supporting_control_memory_copy_len0.wat`
- `supporting_memory_fill_same_shape.wat`
- `supporting_memory_copy_len1.wat`
- `supporting_memory_copy_len64_safe.wat`
- `supporting_memory_copy_src_eq_dst_len32.wat`
- `supporting_memory_copy_src_plus1024_len32_safe.wat`

## Test Case

Primary reproducer loop body:

```wat
(local.get $i)
(i32.wrap_i64)
(i32.const 65504)
(i32.and)
(local.get $i)
(i32.wrap_i64)
(i32.const 1431655765)
(i32.xor)
(i32.const 65504)
(i32.and)
(i32.const 32)
(memory.copy)
```

The reduced reproducer uses:
- trip count: `2^28`
- one page of memory: `(memory 1)`
- both src/dst addresses constrained to a small low-memory window

The closest controls are:
- same shape, but `memory.copy` length changed to `0`
- same shape, but `memory.copy` replaced with `memory.fill`
- same shape, but copy lengths changed across `1/4/8/16/32/64`
- same shape, but src/dst relation changed to `src == dst` and `src = dst + 1024`

## Steps to Reproduce

1. Build the primary testcase:

```bash
wat2wasm primary_reproducer_memory_copy_len32.wat -o primary_reproducer_memory_copy_len32.wasm
```

2. Warm up once:

```bash
wasmtime primary_reproducer_memory_copy_len32.wasm
```

3. Measure runtime:

```bash
perf stat -r 3 -e 'task-clock' wasmtime primary_reproducer_memory_copy_len32.wasm
```

4. For comparison, run the same flow on the supporting testcases listed above.

5. If helpful, compare against Wasmer Cranelift with:

```bash
wasmer run primary_reproducer_memory_copy_len32.wasm
perf stat -r 3 -e 'task-clock' wasmer run primary_reproducer_memory_copy_len32.wasm
```

## Expected and actual Results

### Primary `memory.copy` reproducer and close controls

| testcase | shape | wasmer_cranelift (s) | wasmtime (s) | ratio |
|---|---|---:|---:|---:|
| control_drop | target removed | 0.09570 | 0.08054 | 0.84x |
| memory.copy len=0 | xor-shaped src/dst, bounded window | 0.97108 | 2.61960 | 2.70x |
| memory.copy len=32 | xor-shaped src/dst, bounded window | 0.76792 | 2.68820 | 3.50x |
| memory.fill len=32 | same bounded address shape | 0.64743 | 2.25270 | 3.48x |

Observed pattern:
- the target-removed control is fast in both runtimes;
- Wasmtime is already much slower for `memory.copy len=0`;
- the slowdown remains for `memory.copy len=32`;
- a related bulk-memory instruction (`memory.fill`) shows a similar gap.

This makes the anomaly look more like bulk-memory helper/runtime-path cost than payload movement cost.

### Length sweep for `memory.copy`

| testcase | wasmer_cranelift (s) | wasmtime (s) | ratio |
|---|---:|---:|---:|
| len=0 | 0.97080 | 2.73150 | 2.81x |
| len=1 | 0.97112 | 2.99300 | 3.08x |
| len=4 | 0.89589 | 2.82370 | 3.15x |
| len=8 | 0.89769 | 2.81460 | 3.14x |
| len=16 | 0.91569 | 2.77790 | 3.03x |
| len=32 | 0.76524 | 2.65780 | 3.47x |
| len=64 (safe window) | 0.76253 | 2.68210 | 3.52x |

Observed pattern:
- from `len=0` through `len=64`, the slowdown ratio stays broadly stable;
- the main trigger does not seem to be the payload size itself.

### Src/dst relation sweep for `memory.copy len=32`

| testcase | wasmer_cranelift (s) | wasmtime (s) | ratio |
|---|---:|---:|---:|
| src == dst | 0.75430 | 2.61270 | 3.46x |
| src = dst + 1024 (safe) | 0.72937 | 2.57010 | 3.52x |

Observed pattern:
- the gap remains even when the copy is self-copy or a fixed-offset in-bounds copy;
- this does not look specific to the original xor-shaped address relation;
- this also does not look primarily driven by overlap semantics.

### Family-level consistency

The original full-trip generated `memory_copy_*` seeds all showed `wasmtime > wasmer_cranelift`:

| testcase | wasmer_cranelift (s) | wasmtime (s) | ratio |
|---|---:|---:|---:|
| memory_copy_1 | 12.1567 | 39.8686 | 3.28x |
| memory_copy_2 | 13.4606 | 36.9620 | 2.75x |
| memory_copy_3 | 19.7391 | 36.3320 | 1.84x |
| memory_copy_4 | 23.0015 | 36.1513 | 1.57x |
| memory_copy_5 | 9.9472 | 37.0502 | 3.72x |

Related `memory_fill_*` seeds also showed the same direction:

| testcase | wasmer_cranelift (s) | wasmtime (s) | ratio |
|---|---:|---:|---:|
| memory_fill_1 | 9.8347 | 31.9666 | 3.25x |
| memory_fill_2 | 12.0900 | 32.1992 | 2.66x |
| memory_fill_3 | 12.9405 | 34.5910 | 2.67x |

## Versions and Environment

- wasmtime: 41.0.0 (4898322a4 2025-12-18)
- wasmer: 6.1.0
- WAMR: iwasm 2.4.4
- wasmedge: 0.16.1-18-gc457fe30
- wabt: 1.0.39
- llvm: 21.1.5
- Host OS: Ubuntu 22.04.5 LTS x64
- CPU: 12th Gen Intel® Core™ i7-12700 × 20

## Extra Info

For the primary reduced testcase, I also checked Wasmtime CLIF to make sure the benchmark is still alive.

I generated CLIF with:

```bash
wasmtime compile -C cache=n --emit-clif out_dir primary_reproducer_memory_copy_len32.wasm
```

In the generated CLIF for the hot loop, the operation is still lowered through a per-iteration builtin call equivalent to:

```text
call fn0(vmctx, 0, dst, 0, src, len)
```

The emitted builtin `wasmtime_builtin_memory_copy` still performs a deeper indirect runtime call:

```text
v11 = call_indirect sig0, v10(v0, v1, v2, v3, v4, v5)
```

So this does not look like dead-code elimination or a broken benchmark scaffold.

The strongest trigger condition I can currently support is:
- repeated small bounded bulk-memory operations;
- one-page memory with a hot low-memory window;
- slowdown present for both `memory.copy` and `memory.fill`;
- largely independent of copy length (`0..64` in this sweep) and src/dst relation.

I have not confirmed the internal root cause, so I’m only reporting the measured trigger pattern here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.