BaryoDev / BaryoDev/barakoCMS

An outbound action gets one attempt at the socket, so a transient blip costs a whole durable retry

Open
#703 0 comments 0 reactions 0 assignees View on GitHub
core enhancement
Dominant language
C#
Stars
6
Forks
7
Avg merge
4h 42m
Merged PRs (30d)
307

Description

`WebhookAction` makes a single call and lives with whatever comes back:

```csharp
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, ct);
```

So a connection reset, a 503 from a load balancer rolling pods, or a DNS blip costs the run a full durable attempt. That is one of only five (`WorkflowRetryPolicy.MaxAttempts`), plus thirty seconds or more of backoff before anything is tried again. The same shape is in `RequestAction`, `EmailAction` and the S3 client in `BarakoCMS.Files.S3`.

There are two layers here and only one of them exists:

| Layer | Handles | Where |
| --- | --- | --- |
| Durable retry, across processes and days | a provider that is genuinely down, a node that died | `WorkflowRun`, `JobRecord` |
| In-process resilience, within one attempt | a reset socket, one bad pod, a 503 during a deploy | **missing** |

D15 stays true either way: the queue still owns retry. This sits inside a single attempt rather than around it.

### The other half: nothing stops hammering a provider that is down

`WorkflowActionResult` says the intent out loud:

> Worse, it is load a third party did not ask for on account of a typo.

There is no mechanism behind that. When a provider is down, every tenant's workflows keep calling it independently, each one working through its own five attempts. A circuit breaker per provider is what turns that intent into behaviour.

### Proposal

Wrap the outbound calls in Carom (`Carom`, `Carom.Extensions`, `Carom.Http`): a small bounded retry with decorrelated jitter, a timeout, and a circuit breaker keyed per destination.

### The constraint that has to be designed for, not discovered

**The inner budget must fit well inside the lease.** `WorkflowRetryPolicy.LeaseDuration` is 5 minutes and `Jobs:LeaseSeconds` is 600. If inner retries push a handler past its lease, another node reclaims the attempt and runs it again, and `WorkflowRunner` then discards the first node's outcome on purpose:

> This node ran past its lease.

The visible result is one action performed twice and one outcome recorded. So the inner budget is a configured ceiling checked against the lease, not a default left to add up. Worth an explicit test that the maximum possible inner duration is less than the shortest lease.

Second constraint: the retry must respect `WorkflowActionResult.PermanentFailure` semantics. A malformed URL or a 400 is not worth a second socket, and retrying it inside the attempt reintroduces exactly what `Retryable` exists to prevent.

### Where I checked

`barakoCMS/Features/Workflows/Actions/WebhookAction.cs`, `RequestAction.cs`, `EmailAction.cs`, `barakoCMS/Features/Workflows/WorkflowRunner.cs` (lease handling), `WorkflowActionResult.cs`. `Directory.Packages.props` references no resilience library today. Searched open issues for retry, circuit breaker and resilience; nothing open covers the in-process layer.

Contributor guide

Open the contributing guide

Research direction

Start with WebhookAction.cs, RequestAction.cs, EmailAction.cs, WorkflowActionResult.cs, and WorkflowRunner.cs, then inspect WorkflowRetryPolicy and Jobs:LeaseSeconds. Trace how leases and PermanentFailure/Retryable outcomes are handled before assessing the Carom integration. Done means the bounded in-process policy, per-destination circuit breaker, lease-budget check, and explicit tests for those constraints are defined and covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.