SignalWithStart hangs forever on an orphaned current-execution pointer

Expected Behavior

SignalWithStartWorkflowExecution for a workflow ID makes progress: it either signals the running execution or starts a new run, and returns within the client deadline.

I suppose, but haven't checked, that this behavior could occur in other APIs which interacts with current executions in a similar fashion (Continue As New?, Updates?, ...)

Actual Behavior

We have observed that SignalWithStart API call for one workflow ID to fail consistently with context deadline exceeded error, across multiple calling services, indefinitely. Other workflows are unaffected.

What we found is that that workflow is in a corrupt state in Cassandra: the current-execution pointer row (sentinel permanentRunID) is here, but the mutable state and history of the run it points to are both gone. Reading the rows on the owning shard shows the pointer present, zero mutable-state rows for the referenced run, and zero history nodes. The pointer's decoded execution_state names a run that completed with status TIMED_OUT.

Running an AdminService's DeleteWorkflowExecution API call on the workflow ID / run ID cleared up the dangling row and let the SignalWithStart API call succeed afterwards.

Here is the reasoning of what could happen:

SignalWithStart resolves the current run from the pointer, then loads that run's mutable state. The load returns NotFound, so the call treats it as "no current execution" and takes the brand-new path (CreateWorkflowModeBrandNew). That insert is conditioned on IF NOT EXISTS for the current row, which still exists, so it fails with CurrentWorkflowConditionFailedError. The handler retries, hits the same condition, and loops until the client deadline. No path reconciles a present pointer against a missing target run, so the call never succeeds.

Steps to Reproduce the Problem

1. Create and complete a workflow so a current-execution pointer exists for run X.
2. At the persistence layer, delete the mutable-state row and history nodes for run X, but leave the current-execution pointer row intact.
3. Call SignalWithStartWorkflowExecution for that workflow ID, and observe it retry internally and fail with a deadline.

Specifications

• Version: v1.30.4
• Platform: Self hosted, backed by Cassandra

Additional Context

We suspect the pointer row was resurrected at the persistence layer. Looking at the code it seems workflow deletions removes the pointer before the mutable state and history, so a partial failure would leave the pointer gone, not surviving, and to the best of my knowledge, no code seems to writes the pointer for a closed workflow.

The likely mechanism is Cassandra deleted-row resurrection: the pointer's tombstone expired before every replica compacted it, and a later repair streamed the pre-delete pointer back while the mutable-state and history tombstones stuck. The specific workflow we observed behavior on is over a year old, which is ample time for that to happen.