lightninglabs / lightninglabs/wavelength

actor: wire up dead-letter handling for all actors (durable and otherwise)

Open
#705 1 comment 0 reactions 2 assignees View on GitHub

@bhandras is already working on this.

Since Sep 8, 2026.

actor enhancement gap mailbox P1 safety
Dominant language
Go
Stars
47
Forks
14
Avg merge
2d 6h
Merged PRs (30d)
103

Description

## Summary

We have a full dead-letter **store** but no dead-letter **handling**. When a
durable actor exhausts its `TellRetryPolicy`, the framework writes the message
to the `dead_letters` table and deletes it from the mailbox — and then nothing
ever looks at it again. No actor consumes it, no monitor surfaces it, no metric
or log-at-the-right-level alerts on it, no RPC exposes it, and no path retries
or recovers it. A dead-lettered message is silently and permanently stuck.

This was found while reviewing the OOR per-session refactor, but it is not
OOR-specific — it affects every durable actor in the daemon.

## What exists vs. what's missing

Exists (`db/actordelivery/store_impl.go`):
- `MoveToDeadLetter`, `GetDeadLetter`, `ListDeadLettersByActor`,
`DeleteDeadLetter` over a `dead_letters` table.
- The durable-actor framework calls `MoveToDeadLetter` + `DeleteMessage` when a
Tell's retry policy gives up (`baselib/actor/durable_actor.go` `handleResult`
/ `handleResultInTx`).

Missing:
- No production caller of `ListDeadLetters` / `GetDeadLetter` (only the store
impl and tests reference them).
- No dead-letter office (DLO) actor wired at the actor-system level; OOR's
registry and per-session actors set no `DLO` either. (Note: the `cfg.DLO`
`ActorRef` is a *separate* path — only used by `trySendToDLO` on
`ErrActorTerminated` during `Send`, not by the retry-exhaustion path, which
always lands in the store table.)
- No monitoring, alerting, RPC surface, or operator-facing visibility for
dead-lettered messages.

## Why it matters (concrete OOR cases, generalizes to all actors)

- **Over-cap incoming hint** (`oor/registry.go` `errIncomingAdmissionCapped`):
a `ResolveIncomingTransferRequest` rejected at the concurrency cap is
Nack-retried by the registry's durable mailbox ~5 times (~31s,
`DefaultTellRetryPolicy`) and then dead-lettered. The serverconn ingress
already acked the operator envelope when the Tell durably enqueued, so there
is no transport re-send. Result: under sustained over-cap a legitimate
incoming transfer hint is dropped, the VTXO is never materialized/monitored
locally, and there is no dead-letter handling to recover it.
- More generally, any durable-actor Tell that exhausts its retry policy
(transient dependency failures, a poison message, a bug) vanishes into
`dead_letters` with no operator signal.

For value-bearing subsystems (OOR, rounds, unroll, ledger) a silently
dead-lettered message can mean stuck or invisible funds, hence the `safety`
label.

## Proposed work

1. **Decide the policy per actor class.** Some messages should hard-fail loudly
(poison), some should be parked for manual/automated replay, some are safe to
drop. Make this an explicit choice, not an accident of defaults.
2. **A dead-letter consumer.** Either a DLO actor or a daemon-owned monitor that
periodically `ListDeadLettersByActor`s, surfaces them (structured log at an
appropriate level + metric/health signal), and supports requeue where the
message is idempotent.
3. **Operator visibility.** An RPC / CLI (or at minimum a health/metrics field)
to see dead-letter counts and inspect entries, instead of needing to query
the DB by hand.
4. **Wire it for all durable actors**, with a sane default, and document the
contract in `baselib/actor` docs (when a message dead-letters, who owns it,
how it is recovered).
5. **Non-durable actors:** confirm/define the analogous behavior (currently the
`DLO` ref + `trySendToDLO` on `ErrActorTerminated`), and make it consistent
with the durable policy.

## Acceptance criteria

- [ ] A dead-lettered message for any durable actor is observable (log + metric
/ health surface), not silent.
- [ ] A documented, per-actor-class policy for what dead-lettering means and how
recovery happens.
- [ ] At minimum, an operator-facing way to list/inspect dead letters.
- [ ] Idempotent messages have a defined requeue path.
- [ ] `baselib/actor` docs describe the dead-letter contract.

## References

- `db/actordelivery/store_impl.go` — dead-letter store CRUD.
- `baselib/actor/durable_actor.go` — `handleResult` / `handleResultInTx`
retry-exhaustion -> `MoveToDeadLetter`.
- `oor/registry.go` — `errIncomingAdmissionCapped` over-cap path that
dead-letters with no recovery (motivating case).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.