Spike: replace the hand-rolled outbox, retry and dead letter queue with Wolverine, without touching HTTP
- Dominant language
- C#
- Stars
- 6
- Forks
- 7
- Avg merge
- 4h 42m
- Merged PRs (30d)
- 307
Description
Jeremy Miller (Marten's author) suggested Wolverine.HTTP after the 4.0 announcement. The HTTP half is the wrong half. The durability half is worth a spike.
## What we hand-roll today
`barakoCMS/Infrastructure/Jobs/` and `Features/Workflows/` are, together, a durable message bus on Marten:
| We built | Where |
| :--- | :--- |
| A transactional outbox | `MartenJobStorageProvider`, the enqueue rides the request's own session through `IHttpContextAccessor` |
| Retry with exponential backoff and a cap | `JobOptions`: `MaxAttempts` 5, `BackoffBaseSeconds` 30, `BackoffMaxSeconds` 3600 |
| A dead letter queue | `JobState.DeadLettered`, plus #527 to purge and redact it |
| Lease-based claim with expiry | `Jobs:LeaseSeconds`, so another instance can take an abandoned job |
| Leader election | `pg_advisory_lock` in `ScheduledContentService`, `WorkflowRunRetentionService`, and now `SchemaApplyLock` |
| A background runner | `WorkflowRunner` as a `BackgroundService` |
| Delivery records with retention | `WebhookDelivery` and `WebhookDeliveryRetentionService`, and #671 to give connector requests the same |
| Idempotency keys | 17 files, and #612 says replay returns 409 and keys never expire |
Most of a day's issues in September came out of that surface: #609 and #686 (schema race, the racing party is the job worker), #683 (the jobs table missing from the 4.0 migration), #527, #612, #671, #575, #576.
## What Wolverine gives, verified against its docs
- `.IntegrateWithWolverine()` on `AddMarten(...)` puts the inbox and outbox in this Postgres database.
- `opts.Policies.AutoApplyTransactions()`, where taking `IDocumentSession` opts a handler into a transaction and `IQuerySession` does not. That is the discipline we enforce by hand.
- `opts.OnException().RetryWithCooldown(...).Then.MoveToErrorQueue()`.
- Durable dead letter storage, and `UseDurableLocalQueues()` for locally handled background work.
Licence checked before anything else, because ImageSharp (#626) and FluentAssertions both moved under us: **Wolverine is MIT**, from the repository's own LICENSE file. Clean against MPL-2.0.
We are also already partway in. `Program.cs:115` calls `RunJasperFxCommands(args)` and `Directory.Packages.props` pins `JasperFx 2.56.0` for `db-patch` and `db-assert`.
## Scope: no HTTP change
The endpoints stay FastEndpoints. `CLAUDE.md` section 6 makes the HTTP surface a public contract now that barakoBrew is an external consumer on its own release cadence, and 126 endpoint classes is a lot of wire shape to put in play for no stated problem. Nothing on the board says FastEndpoints is in the way.
The shape to try is Wolverine as a mediator behind the endpoints we already have:
```csharp
// inside an existing FastEndpoints handler
var result = await bus.InvokeAsync(command);
```
Every route, status code and OpenAPI entry is unchanged. barakoBrew sees nothing.
## What the spike has to answer
1. **Does the outbox survive the swap?** Our enqueue commits with the request through the scoped session. Wolverine's outbox does the same thing by a different route. Prove it on one real path, `WebhookAction` delivery, and prove that a failed commit leaves nothing queued. That behaviour has a test today and it must still pass.
2. **What does it add to the database, and to the 3.x upgrade?** New tables in this Postgres. #683 was exactly the failure of adding a table and forgetting the migration, so any answer here includes a migration and a green `scripts/upgrade-check.sh`.
3. **Does it remove the advisory locks or add to them?** Wolverine does its own leader election. Two schemes in one database is worse than one.
4. **Does it fix #686?** The 42P07 race is the job worker touching a document while another host applies the schema. If Wolverine owns the worker lifecycle, does that race go away or move?
5. **Multi-tenancy.** Conjoined tenancy is not optional here. A job carries the tenant of the request that queued it (#528 is open because a background caller cannot name one). Does Wolverine's outbox carry a tenant, and does the handler run in that tenant's session?
6. **What is the module contract cost?** `IBarakoModule` is a public contract under section 6. If modules can queue work, does this change what they compile against?
## Done when
A branch that runs webhook delivery through Wolverine's durable local queue, with the existing tests for transactional enqueue and retry passing unchanged, a migration for whatever it adds, `upgrade-check.sh` green, and a written answer to each question above. Then a decision, not a rollout.
Moved to 4.4.0 on 14 September 2026. The schedule trigger (#839), the publish step with its outbox (#831) and the services registry (#833) all need durable messaging and leader election, so this is decided before they are built rather than after.
Added 14 September 2026, after Jeremy Miller's replies: once the mediator path works, port the public forms endpoints (`GET` and `POST /api/public/forms/{slug}`, the Forms module from #720) to Wolverine.HTTP beside FastEndpoints, and compare lines of code, tests, the OpenAPI document (it must not change), error responses and capability gates. If Wolverine.HTTP is clearly less code with the same contract, new modules start on it. Leader election findings are in #856.
Contributor guide
Research direction
Start with Program.cs:115 and Directory.Packages.props, then inspect barakoCMS/Infrastructure/Jobs/, Features/Workflows/, WebhookAction, and the existing transactional enqueue and retry tests. Run scripts/upgrade-check.sh while tracing the Postgres migration and tenant behavior. Done means a webhook path uses Wolverine's durable local queue, existing tests still pass, the migration is green, and the six stated questions have written answers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, postgresql
- Domain
- backend, databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100