cloudflare / cloudflare/cloudflare-prometheus-exporter
bug: SQLITE_TOOBIG crash when running wrangler dev --local with multiple zones
- Dominant language
- TypeScript
- Stars
- 181
- Forks
- 39
- Avg merge
- 11h 28m
- Merged PRs (30d)
- 3
Description
## Environment
- Running mode: `wrangler dev --local` (Kubernetes sidecar, not deployed to Cloudflare Workers)
- Zones: 10
- wrangler version: bundled in Docker image
## Problem
When running the exporter via `wrangler dev --local`, the `colo-metrics` and `load-balancer-metrics` Durable Objects crash on every alarm cycle with:
```
{"logger":"metric_exporter","namespace":"account::colo-metrics","level":"error","msg":"Refresh failed","error":"string or blob too big: SQLITE_TOOBIG"}
{"logger":"metric_exporter","namespace":"account::load-balancer-metrics","level":"error","msg":"Refresh failed","error":"string or blob too big: SQLITE_TOOBIG"}
```
This repeats every ~60 seconds indefinitely, eventually causing wrangler to crash with:
```
✘ [ERROR] Uncaught Error: Network connection lost.
```
## Root cause
`MetricExporter` stores its entire state as a **single blob** via `ctx.storage.put(STATE_KEY, this.state)`. For `colo-metrics`, the state contains `metrics: MetricDefinition[]` with data for all zones × all PoPs, which grows into a multi-MB JSON blob. miniflare's local SQLite storage has a blob size limit that this exceeds.
`load-balancer-metrics` hits the same issue because it stores per-zone load balancer pool weight data for all zones.
## Impact
- These two DOs never successfully persist state
- Their alarm cycles flood logs with errors
- The accumulated error load eventually causes `Network connection lost` and wrangler crashes
- Pod restarts continuously if not wrapped in a restart loop
## Workarounds
None available without code changes. `METRICS_DENYLIST` only filters Prometheus output — it does not prevent DO creation or data fetching (see related issue).
## Suggested fix
See PR #23 which adds query-name support to `METRICS_DENYLIST`, allowing users to skip DO creation for specific queries entirely. A more complete fix would chunk the stored state across multiple KV keys to handle large datasets.
## Steps to reproduce
1. Deploy with `wrangler dev --local`
2. Configure 5+ Cloudflare zones
3. Observe `colo-metrics` and `load-balancer-metrics` DOs crashing every ~60s
Contributor guide
Research direction
Start with MetricExporter and the ctx.storage.put(STATE_KEY, this.state) call used by the colo-metrics and load-balancer-metrics Durable Objects. Reproduce with wrangler dev --local and 5+ zones, then trace how each object builds its state. Done means alarm cycles persist large datasets without SQLITE_TOOBIG or causing wrangler to lose its network connection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, sqlite, typescript
- Domain
- backend, cloud, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100