feat: background route dispatch with join synchronization

Provenance: AI-drafted (GitHub Copilot CLI / Claude Opus 4.6), human-reviewed and approved, AI-submitted.

Problem

Conductor workflows execute agents strictly sequentially — each agent must complete before the next begins. This is correct for most patterns, but creates unnecessary bottlenecks when independent work could overlap.

Concrete example: In an SDLC workflow, after implementing all tasks in a PR group, the workflow runs a PR lifecycle (reduce → submit → review → merge) that takes 30 minutes. During this time, the next PR group's tasks could start on a different branch. Today they wait.

Existing primitives don't solve this:

• parallel groups are static (all agents defined at YAML time, execute concurrently, no routing)
• for_each groups are homogeneous (same agent, many items)
• Neither supports "start this agent, don't wait, continue with other routing"

Proposed solution: mode: background on routes

A route annotation that starts the target agent/sub-workflow asynchronously:

- name: pr_group_manager
  routes:
    - to: pr_lifecycle
      when: "{{ output.action == 'submit_pr' }}"
      mode: background          # start but don't block
    - to: task_manager
      when: "{{ output.action == 'start_tasks' }}"   # runs immediately
    - to: join_prs
      when: "{{ output.action == 'all_complete' }}"

A new join agent type that waits for all background instances to complete:

- name: join_prs
  type: join
  wait_for: [pr_lifecycle]
  failure_mode: continue_on_error   # fail_fast | continue_on_error | all_or_nothing
  output:
    completed: { type: array }
    errors: { type: array }
    total: { type: number }
  routes:
    - to: pr_finalizer

Semantics

Background dispatch:

• When a route has mode: background, the engine starts the target agent as an asyncio.Task with a frozen context snapshot
• The main loop immediately evaluates the next matching route (first non-background match, or fall-through)
• Multiple background routes can fire from the same agent execution
• Background agents store their outputs in a background-specific context namespace

Join collection:

• wait_for lists agent names that may have been dispatched as background
• The join blocks until all instances of those agents complete
• failure_mode controls error handling (same semantics as for_each):
  • continue_on_error: collect all results, fail only if ALL background tasks failed
  • fail_fast: cancel remaining background tasks on first failure
  • all_or_nothing: wait for all, fail if any failed
• Join output provides completed (successful outputs), errors (failures), and total count

Context isolation:

• Each background agent gets a deep copy of the context at dispatch time
• Background agents cannot see each other's outputs or the main loop's subsequent state
• The join output aggregates results for downstream agents

Why these semantics

• Context snapshots prevent race conditions — no shared mutable state
• continue_on_error default because real workflows have irreversible operations (merged PRs can't be un-merged)
• Join as explicit agent keeps the workflow graph readable — you can see where convergence happens
• Multiple instances are tracked by agent name + dispatch index, similar to for_each item tracking

Design decisions

Decision	Choice	Why
Annotation on routes vs new agent type	Route annotation	Background is a routing concern, not an agent concern — same agent can be foreground or background depending on context
Context sharing	Snapshot at dispatch	Mutable sharing would require locking; snapshots are simpler and match sub-workflow semantics
Join syntax	Explicit agent	Implicit joins (auto-wait at $end) would make control flow invisible
Failure handling	Same as for_each	Proven pattern, users already understand it

Non-goals (v1)

• Background agents seeing each other's outputs (they're isolated)
• Dynamic join (waiting for a computed list of agents)
• Background sub-workflows in for_each (combine later if needed)
• Cancellation of background tasks from the main loop (except via fail_fast on join)

Estimated scope

Schema: Add mode to RouteDef, add JoinDef as new agent type
Engine: Modify execution loop to spawn asyncio tasks for background routes, add join collection logic
Validator: Validate join references, warn on background routes without a downstream join
Events: New event types: background_started, background_completed, background_failed, join_waiting, join_completed
Tests: Background dispatch, join collection, failure modes, context isolation

Start with RouteDef, the execution loop, validator, and event definitions; then review the existing for_each failure handling for comparable behavior. Add background route dispatch, JoinDef collection, validation, events, and tests for dispatch, joins, failure modes, and context isolation. Done means the documented semantics work without shared mutable context.