server: two identical `request_id` submissions racing under `terminate_other` can each terminate the other's run, so one request id answers with two run ids

Observed behavior

Under on_conflict: terminate_other, FlowstateServer.Run starts with WorkflowIDConflictPolicy FAIL when the request carries a request_id, so an exact retry is answered with the run the earlier submission already started rather than terminating it (pkg/flowstate/v1/server/server.go:1560-1613). When that probe reports a conflict the incumbent is not this submission's, the retry re-issues with TERMINATE_EXISTING, which on_conflict: says is the right answer for a genuinely different submission (pkg/flowstate/v1/server/server.go:1626-1646).

Two identical submissions can both take that second path. If both probes land before either terminate-and-start completes, both observe the same earlier incumbent, both classify it as a different submission, and both re-issue with TERMINATE_EXISTING: the first starts run A, the second terminates A and starts B. One request_id is then answered with two run ids, the first caller holds a run id that has been terminated, and whatever steps A ran before termination have already happened.

This predates the FAIL-probe change: on main at 612983c, terminate_other sets TERMINATE_EXISTING up front for every submission, request_id or not (pkg/flowstate/v1/server/server.go:1455-1457 there), so both racing submissions terminate-and-start unconditionally and reach the identical outcome. The probe narrows the window — a submission whose probe lands after the other's run has started is now recognised as a retry and joins it — rather than opening one.

Evidence

The race is between the probe's WorkflowExecutionAlreadyStarted and the re-issue that follows it, both client-side round trips, so it is not reproducible from a unit test without a controllable Temporal frontend. What is verifiable now:

• reusedSubmission decides retry-or-not from the incumbent's recorded submission, which is whatever run held the id at probe time; nothing re-reads it between the decision and the terminate.
• The re-issue at pkg/flowstate/v1/server/server.go:1637 is unconditional — there is no compare-and-terminate, because Temporal's start API does not offer one: WorkflowIDConflictPolicy compares ids, not the submission recorded under them.

Desired outcome

Two concurrent submissions carrying the same request_id answer with the same run id, whatever order their starts interleave in, and neither terminates a run started by the other. request_id's documented promise — one submission, one run — holds under concurrency and not only under retry.

Acceptance criteria

• A test that drives two identical request_id submissions concurrently against a terminate_other workflow and asserts one run id in both responses, with no terminated execution behind either.
• A genuinely different submission still replaces the incumbent, which is what on_conflict: terminate_other means; the fix must not turn a real conflict into a join.
• A submission with no request_id keeps taking the direct TERMINATE_EXISTING path.

Constraints

• Temporal's start call has no conditional-terminate: the policy compares workflow ids, so "terminate only if the incumbent is still the one I probed" has to be built out of something else — an update-with-start, a start-then-verify-then-retry loop with a bounded number of rounds, or a server-side lease keyed by request id.
• Any retry loop needs a bound, per invariant 5, and must not livelock two submissions that keep displacing each other.
• SpecificationAsSubmitted and Reused must stay honest about which submission the named run actually ran; see the arms in Run that already reason about this.

Open questions

• Is the narrow race worth a mechanism, or is the documented answer that concurrent submissions of one request_id are undefined and callers serialize themselves? #1863 records a neighbouring limit of the same promise, and the two may want one answer rather than two.

Start in pkg/flowstate/v1/server/server.go, especially Run and the referenced probe and re-issue arms at lines 1560-1646; compare them with the direct TERMINATE_EXISTING path and the reusedSubmission logic. Review issue #1863 for the neighboring promise boundary before choosing a bounded coordination approach. Done means concurrent identical request_id submissions return one run id, while distinct submissions still replace the incumbent and submissions without request_id retain their current path.