An outbound action gets one attempt at the socket, so a transient blip costs a whole durable retry
- Dominant language
- C#
- Stars
- 6
- Forks
- 7
- Avg merge
- 4h 42m
- Merged PRs (30d)
- 307
Description
`WebhookAction` makes a single call and lives with whatever comes back:
```csharp
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, ct);
```
So a connection reset, a 503 from a load balancer rolling pods, or a DNS blip costs the run a full durable attempt. That is one of only five (`WorkflowRetryPolicy.MaxAttempts`), plus thirty seconds or more of backoff before anything is tried again. The same shape is in `RequestAction`, `EmailAction` and the S3 client in `BarakoCMS.Files.S3`.
There are two layers here and only one of them exists:
| Layer | Handles | Where |
| --- | --- | --- |
| Durable retry, across processes and days | a provider that is genuinely down, a node that died | `WorkflowRun`, `JobRecord` |
| In-process resilience, within one attempt | a reset socket, one bad pod, a 503 during a deploy | **missing** |
D15 stays true either way: the queue still owns retry. This sits inside a single attempt rather than around it.
### The other half: nothing stops hammering a provider that is down
`WorkflowActionResult` says the intent out loud:
> Worse, it is load a third party did not ask for on account of a typo.
There is no mechanism behind that. When a provider is down, every tenant's workflows keep calling it independently, each one working through its own five attempts. A circuit breaker per provider is what turns that intent into behaviour.
### Proposal
Wrap the outbound calls in Carom (`Carom`, `Carom.Extensions`, `Carom.Http`): a small bounded retry with decorrelated jitter, a timeout, and a circuit breaker keyed per destination.
### The constraint that has to be designed for, not discovered
**The inner budget must fit well inside the lease.** `WorkflowRetryPolicy.LeaseDuration` is 5 minutes and `Jobs:LeaseSeconds` is 600. If inner retries push a handler past its lease, another node reclaims the attempt and runs it again, and `WorkflowRunner` then discards the first node's outcome on purpose:
> This node ran past its lease.
The visible result is one action performed twice and one outcome recorded. So the inner budget is a configured ceiling checked against the lease, not a default left to add up. Worth an explicit test that the maximum possible inner duration is less than the shortest lease.
Second constraint: the retry must respect `WorkflowActionResult.PermanentFailure` semantics. A malformed URL or a 400 is not worth a second socket, and retrying it inside the attempt reintroduces exactly what `Retryable` exists to prevent.
### Where I checked
`barakoCMS/Features/Workflows/Actions/WebhookAction.cs`, `RequestAction.cs`, `EmailAction.cs`, `barakoCMS/Features/Workflows/WorkflowRunner.cs` (lease handling), `WorkflowActionResult.cs`. `Directory.Packages.props` references no resilience library today. Searched open issues for retry, circuit breaker and resilience; nothing open covers the in-process layer.
Contributor guide
Research direction
Start with WebhookAction.cs, RequestAction.cs, EmailAction.cs, WorkflowActionResult.cs, and WorkflowRunner.cs, then inspect WorkflowRetryPolicy and Jobs:LeaseSeconds. Trace how leases and PermanentFailure/Retryable outcomes are handled before assessing the Carom integration. Done means the bounded in-process policy, per-destination circuit breaker, lease-budget check, and explicit tests for those constraints are defined and covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100