lightninglabs / lightninglabs/wavelength
actor: wire up dead-letter handling for all actors (durable and otherwise)
@bhandras is already working on this.
Since Sep 8, 2026.
- Dominant language
- Go
- Stars
- 47
- Forks
- 14
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 103
Description
## Summary
We have a full dead-letter **store** but no dead-letter **handling**. When a
durable actor exhausts its `TellRetryPolicy`, the framework writes the message
to the `dead_letters` table and deletes it from the mailbox — and then nothing
ever looks at it again. No actor consumes it, no monitor surfaces it, no metric
or log-at-the-right-level alerts on it, no RPC exposes it, and no path retries
or recovers it. A dead-lettered message is silently and permanently stuck.
This was found while reviewing the OOR per-session refactor, but it is not
OOR-specific — it affects every durable actor in the daemon.
## What exists vs. what's missing
Exists (`db/actordelivery/store_impl.go`):
- `MoveToDeadLetter`, `GetDeadLetter`, `ListDeadLettersByActor`,
`DeleteDeadLetter` over a `dead_letters` table.
- The durable-actor framework calls `MoveToDeadLetter` + `DeleteMessage` when a
Tell's retry policy gives up (`baselib/actor/durable_actor.go` `handleResult`
/ `handleResultInTx`).
Missing:
- No production caller of `ListDeadLetters` / `GetDeadLetter` (only the store
impl and tests reference them).
- No dead-letter office (DLO) actor wired at the actor-system level; OOR's
registry and per-session actors set no `DLO` either. (Note: the `cfg.DLO`
`ActorRef` is a *separate* path — only used by `trySendToDLO` on
`ErrActorTerminated` during `Send`, not by the retry-exhaustion path, which
always lands in the store table.)
- No monitoring, alerting, RPC surface, or operator-facing visibility for
dead-lettered messages.
## Why it matters (concrete OOR cases, generalizes to all actors)
- **Over-cap incoming hint** (`oor/registry.go` `errIncomingAdmissionCapped`):
a `ResolveIncomingTransferRequest` rejected at the concurrency cap is
Nack-retried by the registry's durable mailbox ~5 times (~31s,
`DefaultTellRetryPolicy`) and then dead-lettered. The serverconn ingress
already acked the operator envelope when the Tell durably enqueued, so there
is no transport re-send. Result: under sustained over-cap a legitimate
incoming transfer hint is dropped, the VTXO is never materialized/monitored
locally, and there is no dead-letter handling to recover it.
- More generally, any durable-actor Tell that exhausts its retry policy
(transient dependency failures, a poison message, a bug) vanishes into
`dead_letters` with no operator signal.
For value-bearing subsystems (OOR, rounds, unroll, ledger) a silently
dead-lettered message can mean stuck or invisible funds, hence the `safety`
label.
## Proposed work
1. **Decide the policy per actor class.** Some messages should hard-fail loudly
(poison), some should be parked for manual/automated replay, some are safe to
drop. Make this an explicit choice, not an accident of defaults.
2. **A dead-letter consumer.** Either a DLO actor or a daemon-owned monitor that
periodically `ListDeadLettersByActor`s, surfaces them (structured log at an
appropriate level + metric/health signal), and supports requeue where the
message is idempotent.
3. **Operator visibility.** An RPC / CLI (or at minimum a health/metrics field)
to see dead-letter counts and inspect entries, instead of needing to query
the DB by hand.
4. **Wire it for all durable actors**, with a sane default, and document the
contract in `baselib/actor` docs (when a message dead-letters, who owns it,
how it is recovered).
5. **Non-durable actors:** confirm/define the analogous behavior (currently the
`DLO` ref + `trySendToDLO` on `ErrActorTerminated`), and make it consistent
with the durable policy.
## Acceptance criteria
- [ ] A dead-lettered message for any durable actor is observable (log + metric
/ health surface), not silent.
- [ ] A documented, per-actor-class policy for what dead-lettering means and how
recovery happens.
- [ ] At minimum, an operator-facing way to list/inspect dead letters.
- [ ] Idempotent messages have a defined requeue path.
- [ ] `baselib/actor` docs describe the dead-letter contract.
## References
- `db/actordelivery/store_impl.go` — dead-letter store CRUD.
- `baselib/actor/durable_actor.go` — `handleResult` / `handleResultInTx`
retry-exhaustion -> `MoveToDeadLetter`.
- `oor/registry.go` — `errIncomingAdmissionCapped` over-cap path that
dead-letters with no recovery (motivating case).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.