lightninglabs / lightninglabs/taproot-assets

universe: move federation push off the sync loop

Open
#2,290 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
525
Forks
150
Avg merge
2d 15h
Merged PRs (30d)
31

Description

(N.b., discovered while reviewing #2287. On that branch, such a delay would 'wedge' the watcher's effects outbox, which is worse than on present 'main' strictly because it would delay more stuff. Not a major concern because nothing is lost, but morally it feels like federation push plainly shouldn't be on the sync loop anyway.)

Problem

FederationEnvoy serves three kinds of work from one serial goroutine (syncer(), universe/federation.go:589): the periodic federation sync, single proof pushes, and batch proof pushes. A push request performs the local universe insert, answers the caller, and then pushes the leaves to every federation member, all on that goroutine.

The remote legs carry no deadline. SyncServers and the push tail both run under WithCtxQuitNoTimeout, and the pooled gRPC connections set no keepalive. A member that accepts a stream and never answers therefore holds the loop for as long as the connection stays open.

Callers of UpsertProofLeaf and UpsertProofLeafBatch hop through that loop: they send on an unbuffered channel and wait for the reply, selecting only on the envoy's quit signal (federation.go:908-912). The caller's context is discarded. So a caller whose own work is purely local, the cultivator publishing a confirmed mint batch (tapgarden/cultivator.go:1309), blocks for as long as any remote leg ahead of it is stuck, and no deadline on the caller's side can free it.

Rare in practice, since it needs a live-but-silent member rather than a dead one, but when it happens the batch never confirms until restart, and the exposure spreads to anything else that publishes through the envoy.

Proposal

Take both the local insert and the remote push off the loop:

  • UpsertProofLeaf / UpsertProofLeafBatch perform the local insert on the caller's goroutine, under the caller's context. The archive is already written concurrently from the RPC insert path and both sync paths, so this assumes nothing new.
  • The remote push moves to a single pusher goroutine that drains a FIFO. The loop keeps only the sync tick.
  • Callers see no contract change: the envoy already answers before pushing, so pushes are best-effort from the caller's view today.

Constraints to preserve

  • Ordering. A member rejects a reissuance whose group it has not seen (ErrGroupKeyUnknown), and members learn groups from anchor leaves. The serial loop guarantees anchors reach members before reissuances across batch chunks. One FIFO pusher preserves this; a goroutine per request would not.
  • Backpressure. Callers blocking on the loop is the hang, but it is also the memory bound. A channel either refills the hang once full or grows without bound under a wedge. The push log already keeps one row per leaf and member with attempt counters, and its replay joins the leaf from the local universe. Writing the pending rows on the caller's side and having the pusher drain the log gives durability, restart replay, and no in-memory queue. The log query has no ordering today, so FIFO needs an order by log id.
  • Re-org re-publishes are unlogged. PublishMintProofUpdates builds identifiers with an unspecified proof type, so those pushes never enter the log. A log-driven pusher must either log them or leave them best-effort as they are now.
  • The sync tick stays unbounded. A stuck member still stops federation sync and the pending replay. That is the sync feature's own liveness; moving the replay to the pusher leaves only sync exposed.
  • Tests. universe/federation_test.go drives the push handlers directly and asserts members saw every leaf synchronously. They would await the pusher instead.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with FederationEnvoy.syncer() and the UpsertProofLeaf/UpsertProofLeafBatch paths in universe/federation.go, then inspect the push log and replay behavior. Review the caller in tapgarden/cultivator.go and the direct-handler tests in universe/federation_test.go. Done means local inserts honor caller context, one FIFO pusher preserves ordering and durable replay, and tests await pushed leaves.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.