temporalio / temporalio/temporal

Status check: can the stale mutable state fallback logic in ndc_task_util.go be removed?

Open
#10,718 0 comments 0 reactions 1 assignee View on GitHub

@yycptt is already working on this.

Since Jun 30, 2026.

Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Summary

service/history/ndc_task_util.go (line ~158) contains a TODO
suggesting that fallback logic for stale mutable state can be
removed once the team is confident, based on monitoring the
StaleMutableStateCounter metric:

// TODO: With validateTaskByClock check above, we should never run into 
// the situation where mutable state cache is stale. This is based on 
// the assumption that shard context will never re-acquire the shard 
// after it has been stolen.
// We should monitor the StaleMutableStateCounter metric and remove 
// the logic below once we are confident.

Question

Has the team had a chance to monitor StaleMutableStateCounter
in production since this was written? If the counter has
remained at zero (or near-zero) for a sufficient period, this
might be ready for removal/simplification.

I don't have access to production metrics, so I'm filing this
as a check-in rather than attempting the change myself — removing
this fallback without confidence could introduce correctness
issues.

References

  • service/history/ndc_task_util.go:158-180
  • common/metrics/metric_defs.go:986 (StaleMutableStateCounter definition)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.