ethereum-optimism / ethereum-optimism/optimism
[Flake] op-e2e-fuzz: 'Too long with no output (15m)' — parallel buffers fuzz output until job completion
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
**Claude:** Filed by Claude (AI assistant) on behalf of Adrian Sutton.
## Summary
The `op-e2e-fuzz` CircleCI job intermittently fails with:
```
Too long with no output (exceeded 15m0s): context deadline exceeded
```
The `Fuzz` step is killed by CircleCI's `no_output_timeout: 15m` before producing any output.
## Evidence
- Failing run: https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/127181/workflows/3f6b0990-218b-47ef-9826-bd2df41f008f/jobs/5158358 (job `5158358`)
- Step log (entire output before the kill):
```
parallel --tag -N1 "go test -run NOTAREALTEST -tags cgo_test -v -fuzztime 10s -fuzz {} ./opgeth" ::: FuzzFjordCostFunction FuzzFastLzGethSolidity FuzzFastLzCgo
error: Recipe `fuzz` failed on line 72 with exit code 255
Too long with no output (exceeded 15m0s): context deadline exceeded
```
- Frequency (CircleCI Insights, `main` workflow): **3 failures / 733 runs (~99.6% success)**.
- Job duration profile: median 395s, **p95 824s (~14m)**, max 1476s (~24m) — routinely runs close to the 15-minute no-output timeout.
## Likely Mechanism
`op-e2e/justfile` runs the three fuzz targets under GNU `parallel --tag` (line 72). `parallel` defaults to `--group`, which **buffers each job's output and only flushes it when that job completes**. CircleCI's `no_output_timeout` therefore measures the gap between job completions, not actual test progress.
`FuzzFjordCostFunction` spins up a full op-geth system (`NewOpGeth`) in *every* fuzzing worker process. Under CPU contention on the shared CI box this setup is slow and is not bounded by the test's 60s context (`gethNode.Node.Start()` is not ctx-bounded). When `FuzzFjordCostFunction` finishes more than 15 minutes after the two fast targets, there is a >15-minute window with no output and CircleCI kills the step — even though the test is still making progress.
## Affected
- Package: `op-e2e/opgeth` (`github.com/ethereum-optimism/optimism/op-e2e/opgeth`)
- Recipe: `op-e2e/justfile` `fuzz`
- CI job: `op-e2e-fuzz`
A fix PR is in progress. Root cause and fix will be validated by reproduction before the PR is opened.
Contributor guide
Research direction
Start with the `fuzz` recipe at line 72 of `op-e2e/justfile`, then inspect `FuzzFjordCostFunction` and the other fuzz targets in `op-e2e/opgeth`. Reproduce the `op-e2e-fuzz` CircleCI behavior and confirm that progress output remains visible throughout the run; done means the job no longer hits the 15-minute no-output timeout and the fuzz targets still complete successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, shell
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100