temporalio / temporalio/temporal
NDC conflict resolution silently drops loser-branch non-signal events (e.g. consumed TimerFired)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
I was auditing the history/NDC code paths in v1.9.3 and found what looks like a replay-authority gap on failover. Filing it here with exact locations so it's checkable.
What happens
On NDC conflict resolution the higher failover version wins (service/history/nDCConflictResolver.go:103-120) and the loser state is rebuilt by replaying only the winner branch (service/history/nDCStateRebuilder.go:131-152). Both reapply paths allowlist one event type only:
service/history/nDCEventsReapplier.go:77-86service/history/workflowExecutionContext.go:1133-1140
Only EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED survives. Everything else on the loser branch — including a TimerFired event a worker already consumed — never reappears in authoritative history.
Concrete scenario
Global namespace failover while the demoted branch holds a timer fire the worker already acted on (commands issued, possibly activities dispatched via cluster-local transfer tasks). Post-failover replay authority is the winner branch without the fire: replay diverges from what the worker did, which can surface as nondeterministic-failure loops or duplicated activity side effects if the activity was dispatched pre-failover and re-dispatched after.
Why this seems wrong, not just unknown
The recorded history is the authority workers replay. Here the authority silently diverges from already-acted-upon reality — there is no divergence marker, no failed event, nothing a worker can deterministically reproduce. Compare with how failover-close Failed events are deliberately kept replayable (nDCWorkflow.go:217-247); the loser-branch drop has no equivalent treatment.
Suggestion
Either reapply consumed non-signal facts, or record an explicit, deterministically-reproducible divergence marker — never silent absence. Happy to dig further or test a specific scenario if a maintainer points at the intended invariant here (e.g. if loser-branch consumption is believed impossible, a pointer to the guarantee would close this).
Provenance
Found during a system-level audit assisted by Full-Spectrum Review (evidence-driven audit framework: first-principles necessity checks, business-truth mapping, stable finding IDs). I read the cited code paths myself and verified the mechanism above; the interpretation (impact reachability) is mine.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the cited conflict-resolution and replay paths in service/history/nDCConflictResolver.go, nDCStateRebuilder.go, nDCEventsReapplier.go, and workflowExecutionContext.go, then compare the failover-close handling in nDCWorkflow.go. Reproduce or test the TimerFired scenario and establish whether consumed loser-branch events must remain replayable or require an explicit divergence marker; done means the chosen invariant is deterministic and covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100