cloudflare / cloudflare/workerd

Undocumented "Queue is overloaded. Please back off." error from sendBatch()

Open
#6,053 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.7k
Forks
739
Avg merge
2d 20h
Merged PRs (30d)
174

Description

## Description

`Queue.sendBatch()` intermittently throws an undocumented error:

```
Error: Queue sendBatch failed: Queue is overloaded. Please back off.
```

This error is **not documented** anywhere — not in the [Queues limits](https://developers.cloudflare.com/queues/platform/limits/), [error handling docs](https://developers.cloudflare.com/queues/configuration/batching-retries/), or [changelog](https://developers.cloudflare.com/queues/platform/changelog/). It is distinct from the documented `Too Many Requests` rate limit error.

## Reproduction

- **Context:** `sendBatch()` called inside a Cloudflare Workflow `step.do()` callback
- **Batch size:** 100 messages (~14 msg/s throughput, well under the 5,000 msg/s documented limit)
- **Failure duration:** 57 seconds (request hangs, then fails)
- **Retry behavior:** Succeeds in <1s on immediate retry — clearly transient
- **Frequency:** Observed on multiple occasions across different workflow instances

## Evidence

From `wrangler workflows instances describe`:

```
┌───────────────────────┬───────────────────────┬────────────┬──────────┬──────────────────────────────────────────────────────────────────────┐
│ Start │ End │ Duration │ State │ Error │
├───────────────────────┼───────────────────────┼────────────┼──────────┼──────────────────────────────────────────────────────────────────────┤
│ 2/9/2026, 10:20:57 AM │ 2/9/2026, 10:21:54 AM │ 57 seconds │ ❌ Error │ Error: Queue sendBatch failed: Queue is overloaded. Please back off. │
├───────────────────────┼───────────────────────┼────────────┼──────────┼──────────────────────────────────────────────────────────────────────┤
│ 2/9/2026, 10:22:04 AM │ 2/9/2026, 10:22:05 AM │ 1 second │ ✅ Success│ │
└───────────────────────┴───────────────────────┴────────────┴──────────┘
```

## Root Cause Analysis

From the workerd source (`src/workerd/api/queue.c++`):

```cpp
JSG_REQUIRE(response.statusCode == 200, Error,
kj::str("Queue sendBatch failed: ", response.statusText));
```

The error is the literal HTTP statusText from the internal queue backend. Based on the [Queues v2 architecture blog post](https://blog.cloudflare.com/how-we-built-cloudflare-queues/), the backend uses Storage Shard Durable Objects. The 57-second hang + overload message is consistent with the DO's internal request queue exceeding capacity — likely because the randomly-assigned shard was hot (other tenants' traffic or autoscaling lag).

## Impact

When this error occurs inside a Workflow `step.do()`, the Workflows engine retries the entire callback. If the callback contained `sendBatch()` calls that already succeeded before the failure, **all messages are re-sent with different message IDs** — creating invisible duplicates that cannot be deduplicated at the message level.

In our case, this caused 840 duplicate queue messages per incident.

## Questions

1. **What causes this error?** Is it DO shard overload as described above?
2. **Why does it trigger at ~14 msg/s** when the documented limit is 5,000 msg/s?
3. **Can this be documented** alongside the existing `Too Many Requests` error?
4. **Is there a recommended retry strategy** beyond what the error message suggests?

## Related

- #1483 — `Queue.send()` failing with "Internal Server Error" (similar backend-level failures, open since 2023)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.