microsoft / microsoft/duroxide

Runtime should explicitly handle orphan orchestrator queue messages

Open
#4 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
217
Forks
61
Avg merge
2d 22h
Merged PRs (30d)
3

Description

Problem

When queue messages (e.g., QueueMessage from enqueue_event) arrive in the orchestrator queue before StartOrchestration for a new instance, the runtime's orchestration dispatcher encounters a batch with no instance, no history, and no StartOrchestration/ContinueAsNew message. The current behavior:

  1. fetch_orchestration_item returns the batch with orchestration_name="Unknown"
  2. The runtime logs "completion messages for unstarted instance" and "empty effective batch"
  3. The runtime acks the batch, which permanently deletes the queue rows
  4. The events are lost forever

This was discovered via the sample_config_hot_reload_persistent_events_fs e2e test, which enqueues events before starting an orchestration.

Current Provider-Side Workaround

Both duroxide-pg and duroxide-pg-opt have implemented a provider-side fix in their fetch_orchestration_item stored procedure:

  1. Scan ALL messages for StartOrchestration/ContinueAsNew (not just messages[0]), matching the SQLite provider's work_items.iter().find() behavior
  2. If no start item found: release locks and return nothing, leaving messages in the queue until StartOrchestration arrives

This works but pushes responsibility to the provider, which:

  • Is fragile (providers must each implement this correctly)
  • Cannot add a visible_at delay to prevent tight re-fetching (any delay risks events being lost if the orchestration completes before the delay expires)
  • Relies on LISTEN/NOTIFY for backpressure to prevent tight-looping

Proposed Runtime-Level Fix

The runtime's orchestration dispatcher should handle this case explicitly:

  1. When fetch_orchestration_item returns a batch with no instance and no StartOrchestration/ContinueAsNew in the messages, the runtime should abandon the batch (not ack it)
  2. The abandon should use a reasonable delay (e.g., 500ms) so items become available again later
  3. This keeps the contract simple: providers return whatever is in the queue, and the runtime decides what to do

This would also allow removing the provider-side workarounds.

Affected Code

  • Runtime: dispatchers/orchestration.rs - the "completion messages for unstarted instance" code path
  • Provider trait: abandon_orchestration_item is already available for this purpose

References

  • duroxide-pg-opt migration 0006_fix_orphan_queue_messages.sql
  • duroxide-pg migration 0016_fix_orphan_queue_messages.sql
  • Test: sample_config_hot_reload_persistent_events_fs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the completion messages for unstarted instance path in dispatchers/orchestration.rs and the existing abandon_orchestration_item provider trait method. Run the sample_config_hot_reload_persistent_events_fs test to reproduce the orphan-message behavior. Done means orphan batches are not acknowledged and a regression test verifies queued events survive until orchestration starts.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.