Support for "workflow chain IDs" in all APIs that are able to address the "latest" run

We discussed this over slack. Adding an Issue here so that we can keep track. While this is not (yet) time-critical, I would like to design the .NET SDK under the assumption that this gets eventually implemented before we release production-ready versions of the SDK.

Below, I copy a slightly edited version of the slack conversation for context and records.

x x x x x x x x x x x x

Hey folks, I am trying to understand the feasibility of the following:

Consider the public server APIs that operate on a particular Workflow Run. E.g. QueryWorkflow, TerminateWorkflowExecution, and many others. These APIs tend to in-take WorkflowExecution, which is a tuple of (workflow_id, run_id). Also, for most (all?) such APIs the run_id may be omitted. In such cases the invocation will apply to the most recent run that carries the specified workflow_id.
This is the situation today. Please correct me if I am wrong. 😃

Now two questions (first one may have been asked before).

(1)
Could those APIs be extended such that instead of specifying the run_id, the user could specify the chain_start_run_id (meaning run_id of the first (i.e. the oldest) run in the execution chain)? Then the API would apply to the most recent run (i.e. the newest) in the chain specified by the chain_start_run_id. If the chain finishes at some point and a new chain with the same workflow id is started, then invocations where (run_id, chain_start_run_id) is specified would not "flow" . They would continue to refer to the finished chain.

The purpose of this is hopefully clear: A chain represents a workflow with one or more runs (caused by retries, continue-as-new continuations, ...). Once such a chain finishes, the workflow logically concludes. A new chain is a completely new workflow (with the same workflow id). Typically, a user who interacts with a specific workflow does not want to switch to interacting with a new workflow without noticing.

I assume that the answer to this part of the question is Yes.
There is even a corresponding PR for the API.

( There, chain_start_run_id is called first_execution_run_id, but the name does not matter at this stage. For the current discussion just the concept, not the term is critical, so I'll temporarily stick to chain_start_run_id for brevity/clarity. In fact, I would love to coin the term "workflow chain id", as it is such an important concept that it deserves its own name. But, again, this terminology is not in scope here. 😃 ).

Either way, that PR does not really solve the issue completely. The SDKs need to not only be able to supply the chain_start_run_id, but also they need to know it. Thus:

(2)
Now the second (related) question: Can all those APIs be extended in a way so that their return payload includes the chain_start_run_id of the chain that contains the run to which the call in fact applied?

For example:

A user calls SignalWorkflowExecution(workflowId="W1", runId=null, ...).

This means "send a signal to the latest (=most recent) run with the workflow id "W1"".
Now, the server will determine the latest run with that workflow id and deliver the signal to that particular run. (Lets assume that the run_id of that run was "R42".)

After that, the user probably wants to continue interacting with "the workflow" that was affected by that signal. On a technical level "the workflow" is a chain-of-workflow-runs. They likely want to continue interacting with the "latest" run only as long as the "latest" run is still a part of the same chain as the run "R42" was. If the chain finishes and a new run is started with the same workflow id, that run is no longer part of the same logical workflow. Then the user likely does not want to interact with that run in the same session.

How can we enable the user to avoid unwillingly "overflowing" beyond the end of the workflow chain?

We ensure that SignalWorkflowExecution(..) includes the chain_start_run_id into its return payload. Then, after staring the set of interactions as described above, the client knows the chain_start_run_id of chain that contained the run with run_id="R42". (Let's assume that chain_start_run_id was "R18"). So, all subsequent invocations would include that information.
E.g., to send another signal, the user invokes
SignalWorkflowExecution(workflowId="W1", chain_start_run_id="R18", runId=null, ...)

which means "send signal to the latest run in the chain with chain_start_run_id="R18"".

If the user wanted to address the actually latest run, without restricting the call to the same workflow chain as they interacted previously, they would simply no longer include the chain_start_run_id.

Problem solved. :)

This may sound a little complicated, but I believe once you think it though, it appears quite straight forward. And, of course, we do not actually expect users to deal with the complexity. Language-SDKs will store the chain_start_run_id into whatever object they use to refer to a workflow and to invoke APIs on it (e.g. to send a signal to a workflow).

So: my question to the server team is: how hard / feasible is it to extent the APIs in the manner described? It is something we can reasonably tackle?

Thank you!

x x x x x x x x x x x x

Below is a minimally edited record of the Slack conversation about this topic between a few people.

Start by reviewing the public workflow-run APIs named in the issue, including QueryWorkflow, TerminateWorkflowExecution, and SignalWorkflowExecution, along with the linked API PR 151. Determine how chain_start_run_id would be accepted and returned across APIs that address the latest run. Done means the affected APIs consistently preserve and return the workflow chain identity.