microsoft / microsoft/duroxide

Runtime should explicitly handle orphan orchestrator queue messages

Open
#5 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
217
Forks
61
Avg merge
2d 22h
Merged PRs (30d)
3

Description

Problem

When queue messages (e.g., QueueMessage from enqueue_event) arrive in the orchestrator queue before StartOrchestration for a new instance, the runtime's orchestration dispatcher encounters a batch with no instance, no history, and no StartOrchestration/ContinueAsNew message. The current behavior:

  1. fetch_orchestration_item returns the batch with orchestration_name="Unknown"
  2. The runtime logs "completion messages for unstarted instance" and "empty effective batch"
  3. The runtime acks the batch, which permanently deletes the queue rows
  4. The events are lost forever

This was discovered via the sample_config_hot_reload_persistent_events_fs e2e test, which enqueues events before starting an orchestration.

Current Provider-Side Workaround

Both duroxide-pg and duroxide-pg-opt have implemented a provider-side fix in their fetch_orchestration_item stored procedure:

  1. Scan ALL messages for StartOrchestration/ContinueAsNew (not just messages[0]), matching the SQLite provider's work_items.iter().find() behavior
  2. If no start item found: release locks and return nothing, leaving messages in the queue until StartOrchestration arrives

This works but pushes responsibility to the provider, which:

  • Is fragile (providers must each implement this correctly)
  • Cannot add a visible_at delay to prevent tight re-fetching (any delay risks events being lost if the orchestration completes before the delay expires)
  • Relies on LISTEN/NOTIFY for backpressure to prevent tight-looping

Proposed Runtime-Level Fix

The runtime's orchestration dispatcher should handle this case explicitly:

  1. When fetch_orchestration_item returns a batch with no instance and no StartOrchestration/ContinueAsNew in the messages, the runtime should abandon the batch (not ack it)
  2. The abandon should use a reasonable delay (e.g., 500ms) so items become available again later
  3. This keeps the contract simple: providers return whatever is in the queue, and the runtime decides what to do

This would also allow removing the provider-side workarounds.

Provider Validation Test

This issue was only caught by an e2e sample test (sample_config_hot_reload_persistent_events_fs), not by any provider validation test. A dedicated test should be added to duroxide::provider_validation that:

  1. Enqueues one or more QueueMessage events for an instance before calling start_orchestration
  2. Then starts the orchestration
  3. Verifies that all pre-enqueued events are delivered to the orchestration (not silently dropped)

This would ensure all provider implementations are validated against this scenario without requiring full e2e tests.

Affected Code

  • Runtime: dispatchers/orchestration.rs - the "completion messages for unstarted instance" code path
  • Provider trait: abandon_orchestration_item is already available for this purpose
  • Test suite: duroxide::provider_validation - add orphan message handling test

References

  • duroxide-pg-opt migration 0006_fix_orphan_queue_messages.sql
  • duroxide-pg migration 0016_fix_orphan_queue_messages.sql
  • Test: sample_config_hot_reload_persistent_events_fs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dispatchers/orchestration.rs at the "completion messages for unstarted instance" path, then inspect the existing abandon_orchestration_item provider trait method. Review duroxide::provider_validation and the sample_config_hot_reload_persistent_events_fs test. Done means pre-enqueued QueueMessage events survive until start_orchestration and are delivered without being acknowledged and lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.