Nexus reapply cannot distinguish "CHASM owns this operation" from "no tree owns it"

This is an AI-generated dump of the background for PR #11381, and the work remaining to get things straightened out.

---

Summary

Reset/replication event reapply cannot distinguish "the CHASM tree owns this Nexus operation" from "no tree owns this Nexus operation". Both look identical to cherryPickChasmEvent, because CHASM answers a missing operation with a bare serviceerror.NotFound.

Because reapply is fail-fast, treating the ambiguous answer as an error let a single unappliable event discard an entire replication batch. #11381 works around this by removing the CHASM fallback entirely, which regresses reset reapply for CHASM-owned Nexus operations. This issue tracks the real fix.

Background

Nexus operations can be backed by either the HSM tree or the CHASM tree, and both coexist on the same mutable state. reapplyEvents (service/history/ndc/workflow_resetter.go) routes each event through cherryPickHSMEvent first, then falls back to cherryPickChasmEvent.

Both frameworks register the same nine Nexus event types — components/nexusoperations/events.go for HSM, chasm/lib/workflow/nexus_library.go:26 for CHASM. nexusLibrary is registered unconditionally in chasm/lib/workflow/fx.go, with no rollout flag, so the CHASM registry recognizes all nine types on every server.

reapplyEvents has two callers:

caller	isReset	batch contents
workflow_resetter.go:873 (reset)	true	batch.Events, unfiltered
events_reapplier.go:64 (replication)	false	pre-filtered by shouldReapplyEvent

shouldReapplyEvent (service/history/workflow/util.go:227) returns true for any HSM-registered event type, so Nexus completion events reach reapply on both paths.

The bug

#10986 (b8e986562, "Support CHASM Nexus operations in workflow reset reapply") moved hsm.ErrStateMachineNotFound out of the skip list and into a CHASM fallback: if HSM doesn't have the operation, try CHASM.

That is sound for reset, which guarantees the reapplied batch shares a prefix with the surviving branch. It is not sound for replication, which has no such guarantee. After a handover forks history, a batch can legitimately carry completions for operations that only ever existed on the discarded branch. Those operations are in neither tree.

CHASM answers those with a bare serviceerror.NewNotFoundf("nexus operation not found for scheduled event ID %d", ...) — seven such sites in chasm/lib/workflow/nexus_events.go (lines 123, 153, 185, 219, 257, 295, 333). cherryPickChasmEvent surfaces anything that isn't ErrEventNotCherryPickable as a hard error, and reapplyEvents returns on first error. BackfillWorkflow commits only on a clean return, so one unappliable event discards the whole batch, including completions already applied earlier in the same batch. The affected operation then never completes and its caller hangs.

Diagnosed against an MCN handover failure in TestReconfigureMCNReplicaWithBenchGoOnTestEnv: one Nexus operation's completion was applied at batch index 0 and then discarded 80 times because an unrelated operation at index 8 existed in neither tree.

The reset path is affected too

This is not replication-only, which rules out the tempting narrow fix of gating the fallback on !isReset.

ScheduledEventDefinition.CherryPick returns hsm.ErrNotCherryPickable unconditionally (components/nexusoperations/events.go:32). So resetting to a point before an operation was scheduled leaves that operation out of the rebuilt HSM tree, while its later NexusOperationCompleted is still in the unfiltered reapply batch. Under #10986 that reached CHASM and failed the entire ResetWorkflowExecution — serviceerror.Internal when ChasmEnabled() is false, CHASM's NotFound when it is true.

Reset's shared-prefix guarantee covers operations scheduled before the reset point, not after, so the fallback is unsound on the reset path as well.

Current workaround (#11381)

cherryPickHSMEvent treats hsm.ErrStateMachineNotFound as cherryPickSkipped again instead of routing to CHASM, restoring pre-#10986 behavior.

What this costs: reset reapply of a Nexus completion for a CHASM-tree operation is silently skipped — the bug #10986 set out to fix. This is not reachable in production today, since CHASM Nexus operations are not enabled or rolled out anywhere, but the feature cannot be rolled out until this is fixed properly.

The regression is pinned by TestNexusOperationAsyncCompletion in tests/nexus_workflow_test.go, which asserts RequireNoHistoryEvent(NEXUS_OPERATION_COMPLETED) on the CHASM rail and RequireHistoryEvent on the HSM rail. When this issue is fixed, that assertion becomes RequireHistoryEvent on both rails.

The correct fix

1. CHASM must report a missing operation as skippable, not as an error. The seven serviceerror.NewNotFoundf sites in chasm/lib/workflow/nexus_events.go should return a sentinel the caller can recognize — ErrEventNotCherryPickable, or a new ErrComponentNotFound — so cherryPickChasmEvent can map "I don't own this operation" to cherryPickSkipped while still surfacing genuine failures.

This is the load-bearing change. With it, cherryPickHSMEvent can safely return cherryPickFallback on ErrStateMachineNotFound again: an operation the CHASM tree owns gets applied, and one that neither tree owns is skipped, with no error either way.

2. Consider making the reapply batch tolerant of a single unappliable event. Defense in depth rather than a strict prerequisite — with (1) in place, no error reaches the batch in the first place. Worth weighing separately, since fail-fast over a whole batch is a sharp edge regardless of this specific bug.

3. Two related defects should be fixed at the same time, both currently unreachable only because the fallback is gone:

• cherryPickChasmEvent returns serviceerror.Internal when ChasmEnabled() is false, rather than skipping.
• The completion-token resolution path (HSM StateMachineRef vs CHASM ComponentRef) needs the same HSM/CHASM routing treatment; see the completion handler in nexusoperations.

4. Add coverage for reset-before-schedule. All four reset call sites in tests/nexus_workflow_test.go reset to a WFT after NexusOperationStarted, so the ErrStateMachineNotFound branch never fires in a functional test. The scenario in "The reset path is affected too" above is uncovered end to end.

Referenced from

• service/history/ndc/workflow_resetter.go — the hsm.ErrStateMachineNotFound case in cherryPickHSMEvent
• tests/nexus_workflow_test.go — TestNexusOperationAsyncCompletion, the chasmEnabled branch of the post-reset assertion

Start with cherryPickHSMEvent and cherryPickChasmEvent in service/history/ndc/workflow_resetter.go, then inspect the seven missing-operation sites in chasm/lib/workflow/nexus_events.go. Run TestNexusOperationAsyncCompletion in tests/nexus_workflow_test.go and add coverage for reset-before-schedule. Done means CHASM-owned operations reapply while operations owned by neither tree are skipped without discarding the batch.