cloudflare / cloudflare/cloudflare-prometheus-exporter

bug: SQLITE_TOOBIG crash when running wrangler dev --local with multiple zones

Open
#24 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
181
Forks
39
Avg merge
11h 28m
Merged PRs (30d)
3

Description

## Environment

- Running mode: `wrangler dev --local` (Kubernetes sidecar, not deployed to Cloudflare Workers)
- Zones: 10
- wrangler version: bundled in Docker image

## Problem

When running the exporter via `wrangler dev --local`, the `colo-metrics` and `load-balancer-metrics` Durable Objects crash on every alarm cycle with:

```
{"logger":"metric_exporter","namespace":"account::colo-metrics","level":"error","msg":"Refresh failed","error":"string or blob too big: SQLITE_TOOBIG"}
{"logger":"metric_exporter","namespace":"account::load-balancer-metrics","level":"error","msg":"Refresh failed","error":"string or blob too big: SQLITE_TOOBIG"}
```

This repeats every ~60 seconds indefinitely, eventually causing wrangler to crash with:

```
✘ [ERROR] Uncaught Error: Network connection lost.
```

## Root cause

`MetricExporter` stores its entire state as a **single blob** via `ctx.storage.put(STATE_KEY, this.state)`. For `colo-metrics`, the state contains `metrics: MetricDefinition[]` with data for all zones × all PoPs, which grows into a multi-MB JSON blob. miniflare's local SQLite storage has a blob size limit that this exceeds.

`load-balancer-metrics` hits the same issue because it stores per-zone load balancer pool weight data for all zones.

## Impact

- These two DOs never successfully persist state
- Their alarm cycles flood logs with errors
- The accumulated error load eventually causes `Network connection lost` and wrangler crashes
- Pod restarts continuously if not wrapped in a restart loop

## Workarounds

None available without code changes. `METRICS_DENYLIST` only filters Prometheus output — it does not prevent DO creation or data fetching (see related issue).

## Suggested fix

See PR #23 which adds query-name support to `METRICS_DENYLIST`, allowing users to skip DO creation for specific queries entirely. A more complete fix would chunk the stored state across multiple KV keys to handle large datasets.

## Steps to reproduce

1. Deploy with `wrangler dev --local`
2. Configure 5+ Cloudflare zones
3. Observe `colo-metrics` and `load-balancer-metrics` DOs crashing every ~60s

Contributor guide

Open the contributing guide

Research direction

Start with MetricExporter and the ctx.storage.put(STATE_KEY, this.state) call used by the colo-metrics and load-balancer-metrics Durable Objects. Reproduce with wrangler dev --local and 5+ zones, then trace how each object builds its state. Done means alarm cycles persist large datasets without SQLITE_TOOBIG or causing wrangler to lose its network connection.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, sqlite, typescript
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.