google / google/capsem

P4: Evaluate local Turso crate for session DB write concurrency and async I/O

Open
#85 0 comments 0 reactions 0 assignees View on GitHub
type:bug type:performance
Dominant language
Rust
Stars
72
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
5

Description

## Summary

Evaluate whether replacing or optionally backing `capsem-logger`'s per-session SQLite storage with the local Rust `turso` crate (`tursodatabase/turso`) would improve Capsem's telemetry write path, especially around concurrent writes and async I/O.

This is a low-priority investigation item, not a migration decision.

## Context

Capsem currently writes each VM's `session.db` through a dedicated `DbWriter` thread that owns a `rusqlite` connection, batches queued write ops into transactions, and uses SQLite WAL mode for concurrent readers.

The local Turso crate is an in-process SQLite-compatible database written in Rust. Its advertised advantages include MVCC/`BEGIN CONCURRENT` write behavior and native async support. The question is whether those advantages apply to Capsem's current architecture, where writes are intentionally serialized through one ordered journal writer per VM.

## Questions to answer

- Would Turso improve high-volume telemetry ingestion versus the current `DbWriter` batching model?
- Does native async DB access reduce complexity or latency compared with the existing writer-thread bridge?
- Do Turso's concurrent write benefits matter if Capsem keeps one writer per `session.db`?
- Are there compatibility issues with the current schema, pragmas, read-only inspect paths, WAL/checkpoint behavior, support bundles, and fork/image copying?
- How does Turso behave on macOS, given Capsem's current primary platform?

## Suggested evaluation

- Build a small backend spike behind the `capsem-logger` boundary, not a broad migration.
- Benchmark write throughput, tail latency, read latency during writes, shutdown/checkpoint behavior, and session DB copy/fork behavior.
- Compare against the current `rusqlite` implementation under realistic net/model/MCP/file event mixes.
- Treat this as P4 unless profiling shows the session logger is actually a bottleneck.

Contributor guide

Open the contributing guide

Research direction

Start at the capsem-logger DbWriter implementation and its current rusqlite connection, then review how session databases handle batching, WAL, reads, checkpoints, copying, and forks. Build the suggested small backend spike behind the capsem-logger boundary and compare Turso with rusqlite under representative workloads. Done means reporting throughput, latency, shutdown behavior, compatibility, and macOS results without assuming a migration.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
backend, databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.