google / google/capsem

DB rehydrate fast path requires main-memory session DB inversion

Open
#92 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
72
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
5

Description

## Problem

During the 1.3 DB hot-path work we benchmarked SQLite image-based rehydrate as a possible replacement for row-by-row disk -> memory replay.

The primitive is fast when targeting `main`:

- current row replay rehydrate for 1M rows: ~1.8-2.0s
- `sqlite3_deserialize` from mmap-style image: ~32.5ms
- `sqlite3_serialize` + `sqlite3_deserialize`: ~74.9ms
- aggregate scan after deserialize: ~30-34ms

But this does **not** fit our current layout cleanly.

Current layout:

- `main` = disk DB
- `mem` = attached in-memory hot schema
- writer writes to `mem`, flushes dirty tables to `main`
- readers query `mem` through temp views

Focused experiments showed both `sqlite3_deserialize` and `sqlite3_backup` fail with `SQLITE_CANTOPEN` when targeting the attached memory schema (`mem`), including both the shared-cache memory URI and plain `:memory:`.

The failed production experiment was backed out and `capsem-logger` remained green.

## Required direction

Do not add route projections, query caches, missing-table fallbacks, or route-specific DB helpers.

The real fix is a DB-layer inversion:

- `main` = in-memory hot session ledger
- durable disk DB attached separately
- DB handle still exposes only `ready()`, `query(sql, params)`, and `write(event)`
- routes own query intent only
- DB object owns SQLite connections, memory/disk, batching, flushing, rehydrate, WAL/serialize/FTS5 later
- blob payload storage remains disk-owned/bounded

## Acceptance criteria

- Rehydrate can use SQLite serialize/deserialize on `main` without attached-schema hacks.
- Route code does not learn whether rows are memory or disk backed.
- Existing ledger correctness tests still pass.
- Route latency gates prove hot reads are near-constant under writes.
- DB microbench records disk baseline vs memory-backed implementation.

## Evidence

Committed benchmark support exists in `capsem-logger` DB microbench. The failed attached-schema deserialize/backup attempt was recorded in Sprinty as a finding, not left in production code.

Contributor guide

Open the contributing guide

Research direction

Start with the committed capsem-logger DB microbench and the existing ledger correctness tests, then trace how routes use the DB handle and how the current main/mem layout performs rehydrate. Done means main is the in-memory ledger with the durable database attached separately, route code remains storage-agnostic, correctness and latency gates pass, and disk-versus-memory benchmarks are recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
backend, databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.