Workflow engine: should unsuccessful workflows survive the retention window?
- Dominant language
- C#
- Stars
- 169
- Forks
- 100
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 270
Description
## Today
The retention sweep purges any workflow in a *finished* state older than the retention period (default ~60 days) — and "finished" includes the unsuccessful ones.
`SelectExpiredWorkflowCandidatesCommand` selects `status IN (3, 4, 5, 6, 7)`, which via `PersistentItemStatusMap.Finished` is `Completed`, `Failed`, `Canceled`, `DependencyFailed`, `Abandoned`. There is no per-status retention setting — `RetentionSettings` exposes only `RetentionPeriod`, `BatchSize` and `Interval`.
Two details worth knowing: the clock is `updated_at`, which retries push forward, so it starts at the last attempt rather than at creation; and once every workflow in a collection is gone, the collection row is deleted too.
Consequences:
- A terminally failed workflow is deleted along with the state a resume would need, so recovery becomes impossible. For a failed side-effect workflow that means a lost Altinn Events registration can no longer be re-delivered.
- The failure disappears from every read model. An "instances with errors" view inherits a hard ~60-day horizon, and an unresolved failure ages from red/orange back to green rather than to anything visible.
## The possible pivot
Purge only `Completed` and `Abandoned`; retain `Failed`, `Canceled` and `DependencyFailed` indefinitely.
The semantics line up cleanly: `Completed` succeeded, `Abandoned` is an explicit human write-off, and everything else is unfinished business nobody has adjudicated. `abandon` already exists as the release valve, so retention becomes a consequence of triage rather than of a timer.
Trade-off: unbounded growth if failures are never triaged. Bounded in practice by failures being rare, but it does make an untended backlog permanent instead of self-clearing.
Worth deciding before the admin-facing surfaces are built on top, since they inherit whichever horizon we pick.
Contributor guide
Research direction
Start at SelectExpiredWorkflowCandidatesCommand and trace PersistentItemStatusMap.Finished, RetentionSettings, and the retention sweep. Review how updated_at and collection deletion are handled, then define the chosen retention policy and the behavior that must be covered before admin-facing surfaces depend on it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100