temporalio / temporalio/temporal

Add a dedicated metric for corrupted workflows

Open
#7,637 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
In our environment when there millions of workflow are running, when there are corrupted workflows due to database layer, it's hard to know which workflows are corrupted.

For the status quo of temporal, corrupted workflows are either:

  • Logged & counted in namespace-level metric ( ConditionFailedError , CurrentWorkflowConditionFailedError, WorkflowConditionFailedError)
  • Logged only without workflow info (for DataLoss error)

Temporal already have a protection for this case by trimming history branch to last valid node. However, when corruption happens on database side, trimming history node won't work (we have case where workflows are corrupted because of ConditionFailedError, current execution record is gone from C*)
Describe the solution you'd like
Adding a new metrics (eg: workflow_corrupted) with label of workflowID and runID. This is more efficient to detect problematic workflows rather than running a full db scan using something like tdbg.

I can help to come up with a PR if you guys think it's useful.

Describe alternatives you've considered
Parsing log, which i think need more compute power than proactively expose information in Temporal server itself.

Additional context
N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with common/persistence/execution_manager.go at the linked history-trimming logic, then trace the existing metrics and logging for ConditionFailedError, CurrentWorkflowConditionFailedError, WorkflowConditionFailedError, and DataLoss. Done means corruption cases emit a dedicated workflow_corrupted metric carrying the requested workflow and run identifiers, with coverage for the affected cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.