cloudflare / cloudflare/workerd

Durable Object ordering is not preserved when mixing RPC and fetch calls on the same stub

Open
#6,561 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.7k
Forks
739
Avg merge
2d 20h
Merged PRs (30d)
174

Description

## Summary

When a caller fires `stub.rpc()` and `stub.fetch()` calls interleaved on the same Durable Object stub without awaiting each one, the DO does not process them in send order. Instead, **all fetch calls are processed before all RPC calls**, regardless of the order they were initiated. This violates the expected E-order / actor-model guarantee.

Same-type ordering works correctly — pure RPC or pure fetch calls maintain send order. The bug is exclusively about cross-type ordering.

```js
// Caller sends: rpc-0, fetch-1, rpc-2, fetch-3, rpc-4, fetch-5, ...
//
// DO receives: fetch-1, fetch-3, fetch-5, ..., rpc-0, rpc-2, rpc-4, ...
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
// all fetches first then all RPCs
```

## Reproduction

Repo: https://github.com/threepointone/test-do-rpc-fetch-ordering

Live production endpoint (confirms this is not a local-only issue):

```bash
curl https://test-ordering.threepointone.workers.dev/test-all?n=20
```

The `mixed` result shows `"inOrder": false` with all fetches before all RPCs.

| Test | Production | wrangler dev | vite dev |
|------|-----------|-------------|---------|
| RPC only (N=20) | PASS | PASS | PASS |
| fetch only (N=20) | PASS | PASS | PASS |
| **mixed RPC+fetch (N=20)** | **FAIL** | **FAIL** | **FAIL** |

## Root Cause

`stub.fetch()` and `stub.rpc()` both create a `WorkerInterface` via `ActorChannel::startRequest()`, but they call different methods on it — `request()` for fetch, `customEvent()` for RPC. These two paths reach the DO's `InputGate` (the FIFO queue that serializes incoming events) through a very different number of async hops:

**fetch path** (~1 hop to InputGate):

```
WorkerEntrypoint::request()
→ context.run()
→ InputGate::wait() ← queued here
→ GlobalScope::request()
```

**RPC path** (~4+ hops to InputGate):

```
WorkerEntrypoint::customEvent()
→ JsRpcSessionCustomEvent::run()
→ incomingRequest->delivered()
→ creates EntrypointJsRpcTarget + Cap'n Proto server
→ fulfills capFulfiller
→ co_await donePromise
↓ (Cap'n Proto dispatches the pipelined call)
→ JsRpcTargetBase::call()
→ co_await kj::yield() ← explicit extra yield
→ ctx.run()
→ InputGate::wait() ← queued here
```

Since all operations originate from the same synchronous JS execution, all fetch calls (fewer hops) enqueue at the `InputGate` before any RPC calls (more hops) get there.

The `kj::yield()` in `JsRpcTargetBase::call()` (added in `9cf133b4f` for ExternalPusher ordering) contributes one extra turn, but is not the sole cause — the Cap'n Proto session setup and capability fulfillment already add several turns before it.

## Relevant Code

- [`src/workerd/api/http.c++`](https://github.com/cloudflare/workerd/blob/main/src/workerd/api/http.c%2B%2B) — `Fetcher::getClientForOneCall()` (RPC dispatch via `addTask(worker->customEvent(...))`) vs `fetchImplNoOutputLock()` (fetch dispatch via `asHttpClient()` → `request()`)
- [`src/workerd/api/worker-rpc.c++`](https://github.com/cloudflare/workerd/blob/main/src/workerd/api/worker-rpc.c%2B%2B) — `JsRpcTargetBase::call()` with `co_await kj::yield()` (~line 1031), `JsRpcSessionCustomEvent::run()` (~line 2166)
- [`src/workerd/io/worker-entrypoint.c++`](https://github.com/cloudflare/workerd/blob/main/src/workerd/io/worker-entrypoint.c%2B%2B) — `WorkerEntrypoint::request()` calls `context.run()` directly; `customEvent()` delegates to `event->run()`
- [`src/workerd/io/io-gate.h`](https://github.com/cloudflare/workerd/blob/main/src/workerd/io/io-gate.h) — `InputGate` with FIFO waiter queue

## Possible Fix

The core issue is that `InputGate::wait()` is called at different points in the call chain for fetch vs RPC. One approach: eagerly acquire the `InputGate` position in `JsRpcSessionCustomEvent::run()` (right after `delivered()`, before the Cap'n Proto session setup), then thread the lock through to the first `ctx.run()` call. `IoContext::run()` already accepts a `kj::Maybe` parameter for exactly this pattern (used by `ensureConstructedImpl`). This would not require changing the `CustomEvent::run()` signature or affecting other custom event types.

The `kj::yield()` in `JsRpcTargetBase::call()` and ExternalPusher ordering would be unaffected — the InputGate position is reserved before the yield runs.

I'm happy to send a PR for this if the team agrees with the approach.

---

*The root cause analysis and fix direction were developed in collaboration with Cursor.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.