celestiaorg / celestiaorg/celestia-node

core(multisource): fast-announcer/slow-server source poisons block ingestion — no health tracking / demotion

Open
#5,039 2 comments 0 reactions 0 assignees View on GitHub
needs:triage
Dominant language
Go
Stars
996
Forks
1.1k
Avg merge
1d 6h
Merged PRs (30d)
34

Description

### Context

Since the multi-source block ingestion redesign ([PR](https://github.com/celestiaorg/celestia-node/pull/5018)), `MultiSource` fans new-height announcements from N core endpoints into one channel, and the Listener fetches each block from **the source that announced the height first** (`GetSignedBlockFrom`). Announce speed is used as a per-height freshness signal for "most responsive peer". There is deliberately no fallback inside the fetch: a failed fetch stores nothing, so the duplicate announcement of the same height from another source retries it (resilience is a property of the fan-in).

This works well when announce speed correlates with serve speed. The hole is a source where it doesn't.

### Problem

A source can be the **fastest to announce** (low RTT to the bridge) while being **slow or unable
to serve blocks** (overloaded, rate-limited, flaky transport). Routing follows the announcement,
so every height is first attempted against the bad source:


t=0.0 source Y announces H (always first - lowest RTT)
t=0.05 source X announces H (duplicate, queued)
t=0.0 listener fetches H from Y ...
t=10.0 ... blockFetchTimeout, nothing stored
t=10.0 listener picks up X's duplicate, fetch succeeds in ~0.2s
→ ~10.2s per height at 6s block time
t=6.0 meanwhile Y has already announced H+1 — first again


The per-height retry works, but **the system never learns**: the next height routes to the same
bad source again. The listener is sequential, so each height costs up to `blockFetchTimeout`
(10s) before the healthy duplicate is even read.

Contributor guide

Open the contributing guide

Research direction

Start by reading the multi-source ingestion redesign in PR 5018 and tracing MultiSource announcements through the Listener and GetSignedBlockFrom. Define how block-fetch failures update source health and how demotion changes subsequent routing, then verify that a fast announcer that cannot serve blocks no longer causes every height to wait for blockFetchTimeout before a healthy duplicate is tried.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.