Poison disposition rides the very lease it is trying to escape, so max_attempts cannot terminate a message with an expired lock

Summary

When an orchestration message exceeds max_attempts, the runtime marks it as poison — but the poison marking, the backoff, and the attempt-count increment all commit through an ack keyed on the message's current lock token. If that lease has expired, all three fail together and the message is never terminated.

The result is that the mechanism designed to stop a pathological message cannot stop it in exactly the case where it is most needed. The message is never poisoned, never backed off, and never abandoned with a delay; its lock simply expires and it becomes visible again, so the retry interval collapses to the lock lease — the most aggressive retry the system can produce, applied to the message that most needs to be left alone.

Verified against main at commit cfe0b8c957ef7ede43c6026ebba0052211de1a49.

This is the core-side half of microsoft/duroxide #46. The provider-side defects that produce the expired lease are tracked separately against the PostgreSQL providers. This issue is about the runtime's inability to recover once handed one, which is provider-independent.

Detail

src/runtime/dispatchers/orchestration.rs:549:

if attempt_count > self.options.max_attempts {
    warn!(
        instance = %instance,
        attempt_count = attempt_count,
        max_attempts = self.options.max_attempts,
        "Orchestration message exceeded max attempts, marking as poison"
    );

    self.fail_orchestration_as_poison(&item, lock_token, attempt_count).await;
    return;
}

fail_orchestration_as_poison (line 1216) commits via ack_orchestration_with_changes(lock_token, …) on both its paths — the corrupted-history path at line 1273 and the normal path below it. When lock_token is dead, the ack fails and the poison marking is silently discarded; note both call sites discard the result with let _ = ….

The function already carries an explicit invariant comment about this ack, at line 1238:

// IMPORTANT: The ack below MUST succeed despite corrupted history rows in the DB.

That requirement was reasoned about for the corrupted history path, but not for the expired lease path. The only escape from the retry loop is gated behind the exact thing that is failing.

Observed signature

Two adjacent, uncorrelated log lines repeating indefinitely:

WARN duroxide::runtime::dispatchers::orchestration: Orchestration message exceeded max attempts,
    marking as poison instance=<id> attempt_count=78874 max_attempts=10
WARN duroxide::runtime::dispatchers::orchestration: ack_orchestration_item failed with
    non-retryable error error=ack_orchestration_item: Invalid lock token

attempt_count=78874 against max_attempts=10 shows the poison path being entered tens of thousands of times without ever taking effect. Membership in this trapped set is monotonic: nothing leaves it without manual out-of-band deletion.

Why this belongs in core, not only in a provider

A provider fix stops this fleet from producing expired leases. It does not stop the runtime from being trapped by any provider that ever does — including third-party implementations of the Provider trait. Terminal disposition should not depend on the lease it is escaping.

Note also that the two log lines above are emitted at the same level with no shared correlation, and the second is worded as though it were transient. Together they mean permanently unrecoverable, and nothing in the current output says so.

Suggested direction

Not prescriptive; the shape that follows from the analysis:

1. Make terminal disposition survive a dead lease. The poison path needs a route that does not depend on the current lock token — for example a lease-independent terminal write, or a re-acquire-then-terminate step when the ack fails with an invalid-token error.
2. Do not silently discard the outcome. Both poison acks use let _ = …. A failed poison marking should at minimum be logged as such, and correlated with the "exceeded max attempts" warning that preceded it.
3. Assert the contract in provider validation. A test in src/provider_validation asserting that a lease returned by fetch_orchestration_item is still valid at the moment it is returned would catch the provider-side cause in every provider at once — including implementations not in this organization. The existing suite exercises lock timeouts (src/provider_validation/atomicity.rs, poison_message.rs) but does not assert lease validity at hand-out time.

For contrast, the bundled SQLite provider computes its lease at insert time from the current clock (src/providers/sqlite.rs:831, let locked_until = Self::timestamp_after(lock_timeout);) and takes no blocking lock, so it does not produce this input today. That is a property of that implementation, not a guarantee of the trait.

Related

• microsoft/duroxide #46 — umbrella issue
• Provider-side defects tracked separately against the PostgreSQL providers

Start in src/runtime/dispatchers/orchestration.rs at the max_attempts branch and fail_orchestration_as_poison, then review the related provider-validation tests in src/provider_validation/atomicity.rs and poison_message.rs. Compare both poison paths and their discarded ack results. Done means expired leases cannot trap poison disposition, failures are logged with the preceding warning, and lease validity is covered by validation.