ethereum-optimism / ethereum-optimism/optimism

op-acceptance-tests: CI flakiness while initialization

Open
#17,084 0 comments 0 reactions 0 assignees View on GitHub
A-acceptance-tests
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

https://app.circleci.com/jobs/github/ethereum-optimism/optimism/3776821 shows flaky failures.

All this issues were observed before, but never been tracked. At first glance all issues stems while initialization the tests, before running the actual test logic.

Two buckets of issues:

1. 404 error
```
=== RUN TestFaucetFund
l2.go:389: �[32mINFO �[0m[08-15|10:51:47.932] Loaded devnet keys: 12 L1 system keys, 30 L2 user keys
l2.go:389: �[32mINFO �[0m[08-15|10:51:47.935] Loaded devnet keys: 12 L1 system keys, 30 L2 user keys
assertions.go:369: �[31mERROR�[0m[08-15|10:51:51.176]
assertions.go:369: Error Trace: /home/circleci/project/op-devstack/dsl/eoa.go:117
assertions.go:369: /home/circleci/project/op-devstack/dsl/eoa.go:123
assertions.go:369: /home/circleci/project/op-acceptance-tests/tests/base/faucet_test.go:19
assertions.go:369: Error: Received unexpected error:
assertions.go:369: operation failed permanently after 3 attempts: 404 Not Found: 404 page not found
```
```
=== RUN TestFaucetFund
l2.go:389: �[32mINFO �[0m[08-15|10:51:47.932] Loaded devnet keys: 12 L1 system keys, 30 L2 user keys
l2.go:389: �[32mINFO �[0m[08-15|10:51:47.935] Loaded devnet keys: 12 L1 system keys, 30 L2 user keys
assertions.go:369: �[31mERROR�[0m[08-15|10:51:51.176]
assertions.go:369: Error Trace: /home/circleci/project/op-devstack/dsl/eoa.go:117
assertions.go:369: /home/circleci/project/op-devstack/dsl/eoa.go:123
assertions.go:369: /home/circleci/project/op-acceptance-tests/tests/base/faucet_test.go:19
assertions.go:369: Error: Received unexpected error:
assertions.go:369: operation failed permanently after 3 attempts: 404 Not Found: 404 page not found
assertions.go:369: Test: TestFaucetFund
assertions.go:369: Messages: must lookup balance
assertions.go:369:
--- FAIL: TestFaucetFund (3.25s)
```

2. Waiting for online error
```
=== RUN TestL1ToL2Deposit
l2.go:389: �[32mINFO �[0m[08-15|10:52:39.909] Loaded devnet keys: 12 L1 system keys, 30 L2 user keys
l2.go:389: �[32mINFO �[0m[08-15|10:52:39.911] Loaded devnet keys: 12 L1 system keys, 30 L2 user keys
el.go:61: �[32mINFO �[0m[08-15|10:52:40.415] Waiting for online
el.go:61: �[32mINFO �[0m[08-15|10:52:40.915] Waiting for online
el.go:61: �[32mINFO �[0m[08-15|10:52:41.415] Waiting for online
...
el.go:61: �[32mINFO �[0m[08-15|10:52:48.915] Waiting for online
el.go:61: �[32mINFO �[0m[08-15|10:52:49.415] Waiting for online
assertions.go:369: �[31mERROR�[0m[08-15|10:52:49.915]
assertions.go:369: Error Trace: /home/circleci/project/op-service/testreq/require.go:64
assertions.go:369: /home/circleci/project/op-devstack/dsl/el.go:60
assertions.go:369: /home/circleci/project/op-devstack/dsl/l1_network.go:67
assertions.go:369: /home/circleci/project/op-acceptance-tests/tests/base/deposit/deposit_test.go:27
assertions.go:369: Error: Condition never satisfied
assertions.go:369: Test: TestL1ToL2Deposit
assertions.go:369: Messages: Expected to be online
assertions.go:369:
--- FAIL: TestL1ToL2Deposit (10.01s)
```

Root cause and fix.

Contributor guide

Open the contributing guide

Research direction

Reproduce the CircleCI job and trace the initialization failures through op-devstack/dsl/eoa.go and op-devstack/dsl/el.go, then compare the callers in tests/base/faucet_test.go and tests/base/deposit/deposit_test.go. Determine why the faucet request returns 404 and why the service never becomes online; done means the initialization failures are fixed and these acceptance tests run reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.