DB rehydrate fast path requires main-memory session DB inversion
- Dominant language
- Rust
- Stars
- 72
- Forks
- 13
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 5
Description
## Problem
During the 1.3 DB hot-path work we benchmarked SQLite image-based rehydrate as a possible replacement for row-by-row disk -> memory replay.
The primitive is fast when targeting `main`:
- current row replay rehydrate for 1M rows: ~1.8-2.0s
- `sqlite3_deserialize` from mmap-style image: ~32.5ms
- `sqlite3_serialize` + `sqlite3_deserialize`: ~74.9ms
- aggregate scan after deserialize: ~30-34ms
But this does **not** fit our current layout cleanly.
Current layout:
- `main` = disk DB
- `mem` = attached in-memory hot schema
- writer writes to `mem`, flushes dirty tables to `main`
- readers query `mem` through temp views
Focused experiments showed both `sqlite3_deserialize` and `sqlite3_backup` fail with `SQLITE_CANTOPEN` when targeting the attached memory schema (`mem`), including both the shared-cache memory URI and plain `:memory:`.
The failed production experiment was backed out and `capsem-logger` remained green.
## Required direction
Do not add route projections, query caches, missing-table fallbacks, or route-specific DB helpers.
The real fix is a DB-layer inversion:
- `main` = in-memory hot session ledger
- durable disk DB attached separately
- DB handle still exposes only `ready()`, `query(sql, params)`, and `write(event)`
- routes own query intent only
- DB object owns SQLite connections, memory/disk, batching, flushing, rehydrate, WAL/serialize/FTS5 later
- blob payload storage remains disk-owned/bounded
## Acceptance criteria
- Rehydrate can use SQLite serialize/deserialize on `main` without attached-schema hacks.
- Route code does not learn whether rows are memory or disk backed.
- Existing ledger correctness tests still pass.
- Route latency gates prove hot reads are near-constant under writes.
- DB microbench records disk baseline vs memory-backed implementation.
## Evidence
Committed benchmark support exists in `capsem-logger` DB microbench. The failed attached-schema deserialize/backup attempt was recorded in Sprinty as a finding, not left in production code.
Contributor guide
Research direction
Start with the committed capsem-logger DB microbench and the existing ledger correctness tests, then trace how routes use the DB handle and how the current main/mem layout performs rehydrate. Done means main is the in-memory ledger with the durable database attached separately, route code remains storage-agnostic, correctness and latency gates pass, and disk-versus-memory benchmarks are recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sqlite
- Domain
- backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100