ethereum-optimism / ethereum-optimism/optimism

op-reth: Database integrity error during backfill

Open
#19,639 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

While trying to sync `Ink Sepolia` from scratch using EL sync this happens.

This is using the new storage v2.

Happening in both `1.11.0` and `1.11.3` versions.

op-node version: `v1.16.8`

```2026-03-18T13:31:02.406Z | {"timestamp":"2026-03-18T13:31:02.406221Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.4999%"}}
2026-03-18T13:31:02.961Z | {"timestamp":"2026-03-18T13:31:02.961550Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.5999%"}}
2026-03-18T13:31:03.255Z | {"timestamp":"2026-03-18T13:31:03.255595Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.6999%"}}
2026-03-18T13:31:03.521Z | {"timestamp":"2026-03-18T13:31:03.521489Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.7999%"}}
2026-03-18T13:31:03.787Z | {"timestamp":"2026-03-18T13:31:03.786956Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.8999%"}}
2026-03-18T13:31:04.049Z | {"timestamp":"2026-03-18T13:31:04.049503Z","level":"INFO","fields":{"message":"Collecting indices","progress":"97.9999%"}}
2026-03-18T13:31:04.311Z | {"timestamp":"2026-03-18T13:31:04.310927Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.0999%"}}
2026-03-18T13:31:04.830Z | {"timestamp":"2026-03-18T13:31:04.830787Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.1999%"}}
2026-03-18T13:31:05.084Z | {"timestamp":"2026-03-18T13:31:05.084596Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.2999%"}}
2026-03-18T13:31:05.344Z | {"timestamp":"2026-03-18T13:31:05.343941Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.3999%"}}
2026-03-18T13:31:05.608Z | {"timestamp":"2026-03-18T13:31:05.608193Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.4999%"}}
2026-03-18T13:31:05.878Z | {"timestamp":"2026-03-18T13:31:05.878860Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.5999%"}}
2026-03-18T13:31:06.426Z | {"timestamp":"2026-03-18T13:31:06.426435Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.6999%"}}
2026-03-18T13:31:06.686Z | {"timestamp":"2026-03-18T13:31:06.686157Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.7999%"}}
2026-03-18T13:31:06.941Z | {"timestamp":"2026-03-18T13:31:06.941142Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.8999%"}}
2026-03-18T13:31:07.184Z | {"timestamp":"2026-03-18T13:31:07.183927Z","level":"INFO","fields":{"message":"Collecting indices","progress":"98.9999%"}}
2026-03-18T13:31:07.410Z | {"timestamp":"2026-03-18T13:31:07.410381Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.0999%"}}
2026-03-18T13:31:07.641Z | {"timestamp":"2026-03-18T13:31:07.641222Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.1999%"}}
2026-03-18T13:31:07.864Z | {"timestamp":"2026-03-18T13:31:07.864709Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.2999%"}}
2026-03-18T13:31:09.560Z | {"timestamp":"2026-03-18T13:31:09.560327Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.3999%"}}
2026-03-18T13:31:09.780Z | {"timestamp":"2026-03-18T13:31:09.780150Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.4999%"}}
2026-03-18T13:31:10.014Z | {"timestamp":"2026-03-18T13:31:10.014435Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.5999%"}}
2026-03-18T13:31:10.241Z | {"timestamp":"2026-03-18T13:31:10.241775Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.6999%"}}
2026-03-18T13:31:10.455Z | {"timestamp":"2026-03-18T13:31:10.455555Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.7999%"}}
2026-03-18T13:31:10.682Z | {"timestamp":"2026-03-18T13:31:10.682687Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.8999%"}}
2026-03-18T13:31:10.939Z | {"timestamp":"2026-03-18T13:31:10.938965Z","level":"INFO","fields":{"message":"Collecting indices","progress":"99.9999%"}}
2026-03-18T13:31:11.523Z | {"timestamp":"2026-03-18T13:31:11.522955Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.0999%"}}
2026-03-18T13:31:11.769Z | {"timestamp":"2026-03-18T13:31:11.768986Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.1999%"}}
2026-03-18T13:31:12.010Z | {"timestamp":"2026-03-18T13:31:12.010252Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.2999%"}}
2026-03-18T13:31:12.257Z | {"timestamp":"2026-03-18T13:31:12.257529Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.3999%"}}
2026-03-18T13:31:12.516Z | {"timestamp":"2026-03-18T13:31:12.516769Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.4999%"}}
2026-03-18T13:31:13.102Z | {"timestamp":"2026-03-18T13:31:13.102577Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.5999%"}}
2026-03-18T13:31:13.356Z | {"timestamp":"2026-03-18T13:31:13.356153Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.6999%"}}
2026-03-18T13:31:13.605Z | {"timestamp":"2026-03-18T13:31:13.605277Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.7999%"}}
2026-03-18T13:31:13.873Z | {"timestamp":"2026-03-18T13:31:13.873871Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.8999%"}}
2026-03-18T13:31:14.138Z | {"timestamp":"2026-03-18T13:31:14.138584Z","level":"INFO","fields":{"message":"Collecting indices","progress":"100.9999%"}}
2026-03-18T13:31:14.417Z | {"timestamp":"2026-03-18T13:31:14.417092Z","level":"INFO","fields":{"message":"Collecting indices","progress":"101.0999%"}}
2026-03-18T13:31:14.979Z | {"timestamp":"2026-03-18T13:31:14.979141Z","level":"INFO","fields":{"message":"Collecting indices","progress":"101.1999%"}}
2026-03-18T13:31:15.057Z | {"timestamp":"2026-03-18T13:31:15.057229Z","level":"INFO","fields":{"message":"Loading indices into database"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088289Z","level":"ERROR","fields":{"message":"Stage encountered a fatal error: database integrity error occurred: database integrity error occurred: write operation CursorAppend failed for key \"0000000000000039cd5e8ae05257ce51c473ddd1290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563ffffffffffffffff\" in table StoragesHistory: the given key value is mismatched to the current cursor position (-30418)","stage":"IndexStorageHistory"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088361Z","level":"ERROR","fields":{"message":"backfill sync failed","err":"database integrity error occurred: database integrity error occurred: write operation CursorAppend failed for key \"0000000000000039cd5e8ae05257ce51c473ddd1290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563ffffffffffffffff\" in table StoragesHistory: the given key value is mismatched to the current cursor position (-30418)"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088392Z","level":"ERROR","fields":{"message":"Fatal error in consensus engine"}}
2026-03-18T13:31:16.088Z | {"timestamp":"2026-03-18T13:31:16.088457Z","level":"ERROR","fields":{"message":"shutting down due to error"}}
2026-03-18T13:31:16.088Z | Error: Fatal error in consensus engine
2026-03-18T13:31:16.088Z |
2026-03-18T13:31:16.088Z | Location:
2026-03-18T13:31:16.088Z | /usr/local/cargo/git/checkouts/reth-e231042ee7db3fb7/d6324d6/crates/node/builder/src/launch/engine.rs:324:43```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing an Ink Sepolia EL sync from scratch with storage v2 and inspect the IndexStorageHistory stage writing to StoragesHistory. Read the failure location in crates/node/builder/src/launch/engine.rs:324 and trace the backfill path. Done means the sync completes without the CursorAppend integrity error and the underlying case is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.