blockblaz / blockblaz/zeam

risc0: prove_execution crashes with GPE in xmss_aggregate during CI

Open
#734 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Summary

The risc0 CI prover test crashes with a **General Protection Exception** inside `rec_aggregation::xmss_aggregate` → `prove_execution`. The crash is a raw hardware fault (SIGSEGV/GPE), not a Rust panic.

## Reproduction

- 4 validators, mock chain of 5 blocks
- First block processes fine (slot 1, no attestations to aggregate)
- Crash happens when aggregating attestations for the next block — 0 children, 4 raw XMSS signatures, `log_inv_rate = 2` (`LOG_INV_RATE_PROD`)
- `init_aggregation_bytecode()` succeeds (bytecode loads from cache OK, `pack_extension()` uses SIMD without issue)
- Crash is inside `prove_execution()`, roughly 2.5 seconds into proving

## What we've ruled out

- **NOT AVX-512** — reproduced with `-Ctarget-cpu=x86-64-v3` (AVX2 only, no AVX-512)
- **NOT a Rust panic** — `RUST_BACKTRACE=short` is set but no panic message appears, just a raw GPE
- **NOT bytecode related** — `init_aggregation_bytecode()` and SIMD-based `pack_extension()` succeed during init

## Environment

- GitHub Actions `ubuntu-latest` (x86_64, Linux 6.17)
- Rust stable toolchain
- `rec_aggregation` at rev `fd8814045deb0ef8fcad4c9f4b1250ee33f7dd01`

## Stack trace

```
General protection exception (no address available)
0x6ff8be8 in ??? (???)
0x70003d8 in ??? (???)
0x6ea77ab in ??? (???)
0x6f83545 in ??? (???)
0x6da5400 in ??? (???)
0x6d853ab in ??? (???)
/home/runner/work/zeam/zeam/pkgs/xmss/src/aggregation.zig:139:35: in aggregateSignatures
const agg_sig = xmss_aggregate(
/home/runner/work/zeam/zeam/pkgs/types/src/aggregation.zig:125:37: in aggregate
try xmss.aggregateSignatures(
/home/runner/work/zeam/zeam/pkgs/types/src/block.zig:205:55: in AggregateInnerMap
try aggregation.AggregatedSignatureProof.aggregate(
/home/runner/work/zeam/zeam/pkgs/state-transition/src/mock.zig:306:52: in genMockChain
var proof = try types.aggregateInnerMap(...)
```

The `???` frames are inside the Rust `multisig_glue` static library (`rec_xmss_aggregate` → `prove_execution`).

## Possible causes

- **Stack overflow** — `prove_execution` uses NTT/sumcheck/WHIR evaluation with deep recursion and large temporary buffers. GitHub Actions runners have 8MB default stack. This would produce a SIGSEGV with no address when hitting the stack guard page.
- **Misaligned SIMD memory access** — AVX2 aligned loads (`_mm256_load_si256`) require 32-byte alignment. Unsafe code in the prover could violate this.
- **Other memory corruption** — out-of-bounds access in the polynomial evaluation or proof construction.

## CI run

https://github.com/blockblaz/zeam/actions/runs/24386808177/job/71222511680?pr=733

## Related

- #732 — wraps `xmss_aggregate` FFI in `catch_unwind` (defensive, but won't help here since this is a hardware fault, not a Rust panic)
- #733 — CI workflow changes (AVX-512 portability, but this crash reproduces with AVX2-only)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the CI prover crash with the four-validator mock chain described in the issue. Trace from pkgs/xmss/src/aggregation.zig:139 through prove_execution in the multisig_glue rec_aggregation library at rev fd8814045deb0ef8fcad4c9f4b1250ee33f7dd01, investigating the listed stack, alignment, and memory-corruption possibilities. Done means the aggregation test completes without a hardware fault in the stated CI environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, zig
Domain
cryptography
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.