ethereum-optimism / ethereum-optimism/optimism
op-reth: Database integrity error during backfill
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
While trying to sync `Ink Sepolia` from scratch using EL sync this happens.
This is using the new storage v2.
Happening in both `1.11.0` and `1.11.3` versions.
op-node version: `v1.16.8`
```2026-03-18T13:31:02.406Z | {"timestamp":"2026-03-18T13:31:02.406221Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.4999%"}}
2026-03-18T13:31:02.961Z | {"timestamp":"2026-03-18T13:31:02.961550Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.5999%"}}
2026-03-18T13:31:03.255Z | {"timestamp":"2026-03-18T13:31:03.255595Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.6999%"}}
2026-03-18T13:31:03.521Z | {"timestamp":"2026-03-18T13:31:03.521489Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.7999%"}}
2026-03-18T13:31:03.787Z | {"timestamp":"2026-03-18T13:31:03.786956Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.8999%"}}
2026-03-18T13:31:04.049Z | {"timestamp":"2026-03-18T13:31:04.049503Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.9999%"}}
2026-03-18T13:31:04.311Z | {"timestamp":"2026-03-18T13:31:04.310927Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.0999%"}}
2026-03-18T13:31:04.830Z | {"timestamp":"2026-03-18T13:31:04.830787Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.1999%"}}
2026-03-18T13:31:05.084Z | {"timestamp":"2026-03-18T13:31:05.084596Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.2999%"}}
2026-03-18T13:31:05.344Z | {"timestamp":"2026-03-18T13:31:05.343941Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.3999%"}}
2026-03-18T13:31:05.608Z | {"timestamp":"2026-03-18T13:31:05.608193Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.4999%"}}
2026-03-18T13:31:05.878Z | {"timestamp":"2026-03-18T13:31:05.878860Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.5999%"}}
2026-03-18T13:31:06.426Z | {"timestamp":"2026-03-18T13:31:06.426435Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.6999%"}}
2026-03-18T13:31:06.686Z | {"timestamp":"2026-03-18T13:31:06.686157Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.7999%"}}
2026-03-18T13:31:06.941Z | {"timestamp":"2026-03-18T13:31:06.941142Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.8999%"}}
2026-03-18T13:31:07.184Z | {"timestamp":"2026-03-18T13:31:07.183927Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.9999%"}}
2026-03-18T13:31:07.410Z | {"timestamp":"2026-03-18T13:31:07.410381Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.0999%"}}
2026-03-18T13:31:07.641Z | {"timestamp":"2026-03-18T13:31:07.641222Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.1999%"}}
2026-03-18T13:31:07.864Z | {"timestamp":"2026-03-18T13:31:07.864709Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.2999%"}}
2026-03-18T13:31:09.560Z | {"timestamp":"2026-03-18T13:31:09.560327Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.3999%"}}
2026-03-18T13:31:09.780Z | {"timestamp":"2026-03-18T13:31:09.780150Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.4999%"}}
2026-03-18T13:31:10.014Z | {"timestamp":"2026-03-18T13:31:10.014435Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.5999%"}}
2026-03-18T13:31:10.241Z | {"timestamp":"2026-03-18T13:31:10.241775Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.6999%"}}
2026-03-18T13:31:10.455Z | {"timestamp":"2026-03-18T13:31:10.455555Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.7999%"}}
2026-03-18T13:31:10.682Z | {"timestamp":"2026-03-18T13:31:10.682687Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.8999%"}}
2026-03-18T13:31:10.939Z | {"timestamp":"2026-03-18T13:31:10.938965Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.9999%"}}
2026-03-18T13:31:11.523Z | {"timestamp":"2026-03-18T13:31:11.522955Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.0999%"}}
2026-03-18T13:31:11.769Z | {"timestamp":"2026-03-18T13:31:11.768986Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.1999%"}}
2026-03-18T13:31:12.010Z | {"timestamp":"2026-03-18T13:31:12.010252Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.2999%"}}
2026-03-18T13:31:12.257Z | {"timestamp":"2026-03-18T13:31:12.257529Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.3999%"}}
2026-03-18T13:31:12.516Z | {"timestamp":"2026-03-18T13:31:12.516769Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.4999%"}}
2026-03-18T13:31:13.102Z | {"timestamp":"2026-03-18T13:31:13.102577Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.5999%"}}
2026-03-18T13:31:13.356Z | {"timestamp":"2026-03-18T13:31:13.356153Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.6999%"}}
2026-03-18T13:31:13.605Z | {"timestamp":"2026-03-18T13:31:13.605277Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.7999%"}}
2026-03-18T13:31:13.873Z | {"timestamp":"2026-03-18T13:31:13.873871Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.8999%"}}
2026-03-18T13:31:14.138Z | {"timestamp":"2026-03-18T13:31:14.138584Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.9999%"}}
2026-03-18T13:31:14.417Z | {"timestamp":"2026-03-18T13:31:14.417092Z","level":"INFO","fields":{"message":"Collecting indices","progress":"101.0999%"}}
2026-03-18T13:31:14.979Z | {"timestamp":"2026-03-18T13:31:14.979141Z","level":"INFO","fields":{"message":"Collecting indices","progress":"101.1999%"}}
2026-03-18T13:31:15.057Z | {"timestamp":"2026-03-18T13:31:15.057229Z","level":"INFO","fields":{"message":"Loading indices into database"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088289Z","level":"ERROR","fields":{"message":"Stage encountered a fatal error: database integrity error occurred: database integrity error occurred: write operation CursorAppend failed for key \"0000000000000039cd5e8ae05257ce51c473ddd1290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563ffffffffffffffff\" in table StoragesHistory: the given key value is mismatched to the current cursor position (-30418)","stage":"IndexStorageHistory"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088361Z","level":"ERROR","fields":{"message":"backfill sync failed","err":"database integrity error occurred: database integrity error occurred: write operation CursorAppend failed for key \"0000000000000039cd5e8ae05257ce51c473ddd1290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563ffffffffffffffff\" in table StoragesHistory: the given key value is mismatched to the current cursor position (-30418)"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088392Z","level":"ERROR","fields":{"message":"Fatal error in consensus engine"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088457Z","level":"ERROR","fields":{"message":"shutting down due to error"}}
2026-03-18T13:31:16.088Z | Error: Fatal error in consensus engine
2026-03-18T13:31:16.088Z |
2026-03-18T13:31:16.088Z | Location:
2026-03-18T13:31:16.088Z | /usr/local/cargo/git/checkouts/reth-e231042ee7db3fb7/d6324d6/crates/node/builder/src/launch/engine.rs:324:43```
Contributor guide
Research direction
Start by reproducing an Ink Sepolia EL sync from scratch with storage v2 and inspect the IndexStorageHistory stage writing to StoragesHistory. Read the failure location in crates/node/builder/src/launch/engine.rs:324 and trace the backfill path. Done means the sync completes without the CursorAppend integrity error and the underlying case is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100