microsoft / microsoft/duroxide

Explicit sub-orchestration id collision with a *running* instance hangs the parent silently

Open
#40 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
217
Forks
61
Avg merge
2d 22h
Merged PRs (30d)
3

Description

Summary

When a parent schedules a sub-orchestration with an explicit instance id that already names a running (non-terminal) instance, the parent waits forever. There is no error, no notification, and no log line.

The terminal half of this collision is handled — terminal_collision_notifications in the orchestration dispatcher fails the parent fast when the target id names a terminal instance. The running half has no equivalent.

Colliding instance is… Outcome
terminal parent fails fast ✅
running parent hangs silently

Arguably "running" is the likelier half: two parents racing on the same id is most plausible precisely when the child is long-lived.

Reproduction

Two unrelated parents pass the same explicit id to schedule_sub_orchestration_with_id. No reserved marker, no provider bypass — public API only.

let parent = |ctx: OrchestrationContext, _i: String| async move {
    ctx.schedule_sub_orchestration_with_id("Child", "shared-child", "x").await
};
// Child parks on an external event so it stays Running.
let child = |ctx: OrchestrationContext, _i: String| async move {
    ctx.schedule_wait("Go").await;
    Ok("child-done".to_string())
};

client.start_orchestration("p1", "Parent", "").await?;   // creates "shared-child", parks
client.start_orchestration("p2", "Parent", "").await?;   // same id, still running

Observed:

child status after p1: Running
p2: HUNG (timeout)
p1 final: Completed { output: "ok:child-done" }
p2 final: Running          <- forever

p2's StartOrchestration is absorbed as a duplicate start for an existing instance. The child eventually completes and notifies only p1 — its recorded parent. p2 gets nothing.

Why it happens

terminal_collision_notifications is only invoked from the terminal fast-ack path in process_orchestration_item:

if temp_history_mgr.is_completed || temp_history_mgr.is_failed || (…CAN…) {
    let orchestrator_items = self.terminal_collision_notifications(&item).await;
    …
}

A colliding start against a running instance never reaches that branch. It falls through to normal processing, where it is deduplicated against the existing history and silently dropped.

Scope

Only reachable when the child id is caller-chosenschedule_sub_orchestration_with_id, schedule_orchestration (detached), or an id already created by Client::start_orchestration. Auto-generated ids cannot collide: they embed both the parent instance id and its execution.

Pre-existing; not introduced by #28. Splitting out so that PR stays scoped.

Possible directions

  1. Detect in the normal path — when an incoming StartOrchestration targets an instance whose recorded parent link differs from the incoming one, notify the incoming parent with SubOrchFailed. Symmetric with the terminal case, but needs care to avoid the false-positive class that the terminal-path parent comparison originally had (see #28 discussion).
  2. Make it loud instead of silent — cheapest option. Log an error whenever a StartOrchestration is discarded for an already-existing instance. Fixes no hang but removes the silence.
  3. Reject at schedule time — not possible in general; the scheduling parent has no way to know the id is taken without a provider read.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in process_orchestration_item and compare the normal StartOrchestration path with terminal_collision_notifications. Reproduce the shared-child case using schedule_sub_orchestration_with_id, then inspect how the existing history and recorded parent link are handled. Done means the running collision no longer leaves p2 silently waiting, with behavior and tests aligned to the chosen direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.