cloudflare / cloudflare/workerd

Durable Object facets with SQLite prevent parent DO from hibernating — causing unexpected duration billing

Open
#6,800 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
8.7k
Forks
739
Avg merge
2d 20h
Merged PRs (30d)
174

Description

### Summary

When a parent DO creates facets that use SQLite storage, the parent DO cannot enter hibernatable state after all requests complete. It stays in "idle, non-hibernatable" state for 70-140 seconds until forced eviction, accumulating duration billing continuously.

Manually calling `ctx.facets.abort()` after each facet operation restores normal hibernation, but this is an expensive workaround (full constructor re-execution on next access).

### Environment

- Runtime: Cloudflare Workers (production)
- Feature: Durable Object Facets (beta)
- Facet usage: Each facet has its own SQLite database (via `drizzle-orm/durable-sqlite`), runs `migrate()` in `blockConcurrencyWhile`, and processes RPC calls

### Reproduction Data

We built a workflow engine where Instance DO (parent) creates Execution facets. Each trigger creates 1 parent + 3 child facets, all complete within ~2 seconds. We measured GB-sec billing across 4 scenarios:

| Scenario | Facets | SQLite | abort() after use | GB-sec / trigger | Wall time / trigger |
|----------|--------|--------|-------------------|-----------------|-------------------|
| A. No facets | 0 | — | — | 0.01 | ~0.08s |
| B. Empty facet (no SQLite) | 1 | No | No | 0.08 | ~0.6s |
| C. Facet + SQLite + migrate | 4 | Yes | No | ~1.0 | ~70-140s (until eviction) |
| D. Facet + SQLite + abort() | 4 | Yes | Yes | 0.41 | ~3.2s |

**Key observation**: Scenario C shows the parent DO staying alive 70-140 seconds after all work completes (zero CPU, zero active requests). Scenario D (identical code, just adding `ctx.facets.abort()`) restores normal ~3 second lifetime.

### Scale of Impact

Over ~27,000 triggers at 5-second intervals:

| Version | Triggers | GB-sec | Per trigger | Error rate |
|---------|----------|--------|-------------|------------|
| Without abort | 27,347 | 9,000 | 0.33* | 28.6% |
| With abort | 326 | 133 | 0.41 | 0.008% |

*The 0.33 number was with a different architecture that had abort via signal routing. When we refactored and accidentally lost the abort calls, billing jumped to ~1.0 GB-sec/trigger.

### Expected Behavior

After all RPC calls to facets complete and no active requests/WebSockets/timers remain, the parent DO should be eligible for hibernation within the normal 10-second inactivity window — regardless of whether facets used SQLite storage.

### Actual Behavior

The parent DO remains in "idle, in-memory, non-hibernatable" state. Based on workerd source analysis:

- `hasClients()` ([server.c++ L2505-L2513](https://github.com/cloudflare/workerd/blob/main/src/workerd/server/server.c++)) recursively checks all facets
- `handleShutdown()` requires `hasClients() === false` to proceed with hibernation
- Something about facets with SQLite causes `hasClients()` (or `isShared()`) to return `true` even when the facet has no active requests

### Current Workaround

We call `ctx.facets.abort(name)` after every facet RPC completes:

```typescript
const output = await this.getExecutionStub(executionId).advance({ input, graph });
this.abortExecution(executionId); // immediately release facet

// abortExecution implementation:
abortExecution(executionId: string): void {
try {
(this.ctx as any).facets.abort(executionId, new Error("FACET_COMPLETED"));
} catch {}
}
```

This works but has drawbacks:
- Next `facets.get()` triggers full reconstruction (constructor + `blockConcurrencyWhile` + `migrate()` + initialization RPCs)
- ~50-700ms overhead per reconstruction
- Must track which facets to abort vs. which are still actively suspended (waiting for external signals/timeouts)

### Questions

1. **Is this expected behavior or a bug?** Should facets with SQLite storage automatically release their "active client" status when idle?

2. **Is there a lighter-weight cleanup mechanism than `abort()`?** Something like `facets.release(name)` that marks the facet as hibernation-eligible without destroying the JS isolate — so the next `facets.get()` returns the existing instance instead of reconstructing?

3. **How are facets billed for duration?** Do facets share the parent DO's 128MB allocation, or does each facet have independent 128MB? Our measurements suggest shared billing (`0.41 GB-sec ÷ 0.128 GB = 3.2s`, matching actual execution time for 1 parent + 3 facets), but documentation doesn't clarify this.

4. **Will [workerd #6087](https://github.com/cloudflare/workerd/issues/6087) (Hibernatable RPC Targets) address this?** Or is the facet + SQLite hibernation issue a separate concern?

### Related Issues

- [workerd #6087](https://github.com/cloudflare/workerd/issues/6087) — Hibernatable RPC Targets
- [workerd #6702](https://github.com/cloudflare/workerd/issues/6702) — Facets + WebSocket cross-DO I/O inconsistency
- [cloudflare/agents #1261](https://github.com/cloudflare/agents/issues/1261) — RPC session pins DO in memory ~120s

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.