temporalio / temporalio/temporal

NDC conflict resolution silently drops loser-branch non-signal events (e.g. consumed TimerFired)

Open
#11,932 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

I was auditing the history/NDC code paths in v1.9.3 and found what looks like a replay-authority gap on failover. Filing it here with exact locations so it's checkable.

What happens

On NDC conflict resolution the higher failover version wins (service/history/nDCConflictResolver.go:103-120) and the loser state is rebuilt by replaying only the winner branch (service/history/nDCStateRebuilder.go:131-152). Both reapply paths allowlist one event type only:

  • service/history/nDCEventsReapplier.go:77-86
  • service/history/workflowExecutionContext.go:1133-1140

Only EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED survives. Everything else on the loser branch — including a TimerFired event a worker already consumed — never reappears in authoritative history.

Concrete scenario

Global namespace failover while the demoted branch holds a timer fire the worker already acted on (commands issued, possibly activities dispatched via cluster-local transfer tasks). Post-failover replay authority is the winner branch without the fire: replay diverges from what the worker did, which can surface as nondeterministic-failure loops or duplicated activity side effects if the activity was dispatched pre-failover and re-dispatched after.

Why this seems wrong, not just unknown

The recorded history is the authority workers replay. Here the authority silently diverges from already-acted-upon reality — there is no divergence marker, no failed event, nothing a worker can deterministically reproduce. Compare with how failover-close Failed events are deliberately kept replayable (nDCWorkflow.go:217-247); the loser-branch drop has no equivalent treatment.

Suggestion

Either reapply consumed non-signal facts, or record an explicit, deterministically-reproducible divergence marker — never silent absence. Happy to dig further or test a specific scenario if a maintainer points at the intended invariant here (e.g. if loser-branch consumption is believed impossible, a pointer to the guarantee would close this).

Provenance

Found during a system-level audit assisted by Full-Spectrum Review (evidence-driven audit framework: first-principles necessity checks, business-truth mapping, stable finding IDs). I read the cited code paths myself and verified the mechanism above; the interpretation (impact reachability) is mine.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the cited conflict-resolution and replay paths in service/history/nDCConflictResolver.go, nDCStateRebuilder.go, nDCEventsReapplier.go, and workflowExecutionContext.go, then compare the failover-close handling in nDCWorkflow.go. Reproduce or test the TimerFired scenario and establish whether consumed loser-branch events must remain replayable or require an explicit divergence marker; done means the chosen invariant is deterministic and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.