ethereum-optimism / ethereum-optimism/optimism

[Flake] op-e2e-fuzz: 'Too long with no output (15m)' — parallel buffers fuzz output until job completion

Open
#21,191 4 comments 0 reactions 0 assignees View on GitHub
C-flake
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

**Claude:** Filed by Claude (AI assistant) on behalf of Adrian Sutton.

## Summary

The `op-e2e-fuzz` CircleCI job intermittently fails with:

```
Too long with no output (exceeded 15m0s): context deadline exceeded
```

The `Fuzz` step is killed by CircleCI's `no_output_timeout: 15m` before producing any output.

## Evidence

- Failing run: https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/127181/workflows/3f6b0990-218b-47ef-9826-bd2df41f008f/jobs/5158358 (job `5158358`)
- Step log (entire output before the kill):
```
parallel --tag -N1 "go test -run NOTAREALTEST -tags cgo_test -v -fuzztime 10s -fuzz {} ./opgeth" ::: FuzzFjordCostFunction FuzzFastLzGethSolidity FuzzFastLzCgo
error: Recipe `fuzz` failed on line 72 with exit code 255
Too long with no output (exceeded 15m0s): context deadline exceeded
```
- Frequency (CircleCI Insights, `main` workflow): **3 failures / 733 runs (~99.6% success)**.
- Job duration profile: median 395s, **p95 824s (~14m)**, max 1476s (~24m) — routinely runs close to the 15-minute no-output timeout.

## Likely Mechanism

`op-e2e/justfile` runs the three fuzz targets under GNU `parallel --tag` (line 72). `parallel` defaults to `--group`, which **buffers each job's output and only flushes it when that job completes**. CircleCI's `no_output_timeout` therefore measures the gap between job completions, not actual test progress.

`FuzzFjordCostFunction` spins up a full op-geth system (`NewOpGeth`) in *every* fuzzing worker process. Under CPU contention on the shared CI box this setup is slow and is not bounded by the test's 60s context (`gethNode.Node.Start()` is not ctx-bounded). When `FuzzFjordCostFunction` finishes more than 15 minutes after the two fast targets, there is a >15-minute window with no output and CircleCI kills the step — even though the test is still making progress.

## Affected
- Package: `op-e2e/opgeth` (`github.com/ethereum-optimism/optimism/op-e2e/opgeth`)
- Recipe: `op-e2e/justfile` `fuzz`
- CI job: `op-e2e-fuzz`

A fix PR is in progress. Root cause and fix will be validated by reproduction before the PR is opened.

Contributor guide

Open the contributing guide

Research direction

Start with the `fuzz` recipe at line 72 of `op-e2e/justfile`, then inspect `FuzzFjordCostFunction` and the other fuzz targets in `op-e2e/opgeth`. Reproduce the `op-e2e-fuzz` CircleCI behavior and confirm that progress output remains visible throughout the run; done means the job no longer hits the 15-minute no-output timeout and the fuzz targets still complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, shell
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.