P4: Evaluate local Turso crate for session DB write concurrency and async I/O
- Dominant language
- Rust
- Stars
- 72
- Forks
- 13
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 5
Description
## Summary
Evaluate whether replacing or optionally backing `capsem-logger`'s per-session SQLite storage with the local Rust `turso` crate (`tursodatabase/turso`) would improve Capsem's telemetry write path, especially around concurrent writes and async I/O.
This is a low-priority investigation item, not a migration decision.
## Context
Capsem currently writes each VM's `session.db` through a dedicated `DbWriter` thread that owns a `rusqlite` connection, batches queued write ops into transactions, and uses SQLite WAL mode for concurrent readers.
The local Turso crate is an in-process SQLite-compatible database written in Rust. Its advertised advantages include MVCC/`BEGIN CONCURRENT` write behavior and native async support. The question is whether those advantages apply to Capsem's current architecture, where writes are intentionally serialized through one ordered journal writer per VM.
## Questions to answer
- Would Turso improve high-volume telemetry ingestion versus the current `DbWriter` batching model?
- Does native async DB access reduce complexity or latency compared with the existing writer-thread bridge?
- Do Turso's concurrent write benefits matter if Capsem keeps one writer per `session.db`?
- Are there compatibility issues with the current schema, pragmas, read-only inspect paths, WAL/checkpoint behavior, support bundles, and fork/image copying?
- How does Turso behave on macOS, given Capsem's current primary platform?
## Suggested evaluation
- Build a small backend spike behind the `capsem-logger` boundary, not a broad migration.
- Benchmark write throughput, tail latency, read latency during writes, shutdown/checkpoint behavior, and session DB copy/fork behavior.
- Compare against the current `rusqlite` implementation under realistic net/model/MCP/file event mixes.
- Treat this as P4 unless profiling shows the session logger is actually a bottleneck.
Contributor guide
Research direction
Start at the capsem-logger DbWriter implementation and its current rusqlite connection, then review how session databases handle batching, WAL, reads, checkpoints, copying, and forks. Build the suggested small backend spike behind the capsem-logger boundary and compare Turso with rusqlite under representative workloads. Done means reporting throughput, latency, shutdown behavior, compatibility, and macOS results without assuming a migration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sqlite
- Domain
- backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100