oasisprotocol / oasisprotocol/oasis-core

go/worker/storage/committee: Explicit error for missing runtime block header

Open
#6,403 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
369
Forks
151
Avg merge
1d 3h
Merged PRs (30d)
6

Description

Problem

Example error:

000000000000000000000000000000000000000000000000e199119c992377cb): failed to get block for round 10240341 (current round: 10365418): roothash: block not found","level":"error","module":"worker/storage","msg":"worker stopped","ts":"2025-11-14T09:33:22.7743955Z"}

This happens when your runtime's State DB latest round is older than the runtime's light history last retained round, which results in the worker fetching the light history header it does not have.

So far I have seen two situations in practice:

  1. You restore consensus from the backup that is 3 months old, and set the pruning to two weeks. Later you also decide to add runtime to it and again you also restore it from the 3 months old snapshot, however you forget to restore 3 months old runtime light history as well. History reindex would than manually reindex consensus, but as you have pruning set to two weeks, you will only reindex last 2 weeks.
  2. People remove consensus state (due to corruption) and do consensus checkpoint sync, but keep the runtime state whose latest state is older than the consensus checkpoint they just restored from. Again this would produce a gap in the light history.

Solution

Make this error log more explicit, and suggest how to intervene manually.

Optimal solutions

  1. Databases should be initialized before starting any workers, and their data sanity checked for any corruption before starting any workers, preventing this error in the first place.
    • Obviously you still stop the node and print what is wrong.
  2. New runtime p2p protocol for light headers, fetch in reverse if missing reindex gap.

Alternative solution

We could also remove runtime state up to the last retained light history round. I don't like this:

  1. We should avoid clearing state without operators consents.
  2. If the runtime state is big this won't work due to #6334.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the worker/storage module by locating the path that logs the missing block error for a runtime light-history header. Make the error explain the state and light-history gap and suggest stopping the node for manual intervention, then verify the resulting message against the two restoration scenarios described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.