temporalio / temporalio/temporal
Add a dedicated metric for corrupted workflows
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Is your feature request related to a problem? Please describe.
In our environment when there millions of workflow are running, when there are corrupted workflows due to database layer, it's hard to know which workflows are corrupted.
For the status quo of temporal, corrupted workflows are either:
- Logged & counted in namespace-level metric (
ConditionFailedError,CurrentWorkflowConditionFailedError,WorkflowConditionFailedError) - Logged only without workflow info (for
DataLosserror)
Temporal already have a protection for this case by trimming history branch to last valid node. However, when corruption happens on database side, trimming history node won't work (we have case where workflows are corrupted because of ConditionFailedError, current execution record is gone from C*)
Describe the solution you'd like
Adding a new metrics (eg: workflow_corrupted) with label of workflowID and runID. This is more efficient to detect problematic workflows rather than running a full db scan using something like tdbg.
I can help to come up with a PR if you guys think it's useful.
Describe alternatives you've considered
Parsing log, which i think need more compute power than proactively expose information in Temporal server itself.
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with common/persistence/execution_manager.go at the linked history-trimming logic, then trace the existing metrics and logging for ConditionFailedError, CurrentWorkflowConditionFailedError, WorkflowConditionFailedError, and DataLoss. Done means corruption cases emit a dedicated workflow_corrupted metric carrying the requested workflow and run identifiers, with coverage for the affected cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100