google / google/capsem

High RAM usage due to in-memory SQLite database

Open Beginner friendly
#213 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
72
Forks
13
Avg merge
1d 2h
Merged PRs (30d)
5

Description

I noticed Capsem could take a lot of RAM after running a long time, which could make my machine unresponsive. It seems that this is coming from a large SQLite database loaded in memory even though it seems that it could be kept on disk. Below is an AI-generated bug report with a fix suggesting to add `security_rule_events` and `security_decision_events` to `DISK_ONLY_TABLES` (which already has `event_body_blobs`).

---

*AI-generated below*

### Summary
When a persistent sandbox runs workloads involving many file or network operations (such as benchmarks or builds), `session.db` accumulates large volumes of `security_rule_events` and `security_decision_events` (often hundreds of thousands of rows totaling 15–25+ GiB).

On service startup, `capsem-service` hydrates database handles for persistent sandboxes via `DbReader::open`, which unconditionally copies all non-`DISK_ONLY_TABLES` from disk into an in-memory SQLite database (`mem`). Because only `event_body_blobs` is currently marked disk-only, `capsem-service` copies all serialized security event JSON payloads into RAM, resulting in:
1. **Multi-minute startup freezes**: A 22 GiB database logged a **165-second** freeze during `register_session_db_handle`.
2. **Process memory exhaustion**: Service RSS balloons to 40–94+ GiB.
3. **Host thrashing and OOM kills**: Drives heavy swap thrashing, triggers `systemd-oomd`, and can render the host unresponsive.

---

### Root Cause
1. In `crates/capsem-service/src/session_db_handles.rs`, `ServiceState::register_session_db_handle` calls `capsem_logger::DbHandle::open_external_reader`.
2. In `crates/capsem-logger/src/reader.rs` (`DbReader::open`), the reader attempts to populate an in-memory mirror:
```rust
schema::with_memory_schema_lock(|| {
schema::create_memory_tables(&conn, &memory_uri)?;
schema::rehydrate_memory_tables_from_disk_once(&conn, schema::hot_ledger_tables())?;
schema::create_memory_read_views(&conn)
})?;
```
3. In `crates/capsem-logger/src/schema.rs`, `hot_ledger_tables()` selects all schema tables except `DISK_ONLY_TABLES`:
```rust
const DISK_ONLY_TABLES: &[&str] = &["event_body_blobs"];
```
4. In `crates/capsem-logger/src/schema/memory_sync.rs` (`sync_memory_tables_from_disk`), it executes:
```sql
INSERT OR REPLACE INTO mem.{table} SELECT * FROM main.{table};
```
For `security_rule_events` (13.2 GiB) and `security_decision_events` (4.5 GiB), this reads gigabytes of records off disk and allocates them directly into the process heap.

---

### Suggested Fix

Mark `security_rule_events` and `security_decision_events` as disk-only tables. Like `event_body_blobs`, read queries will query the disk database directly using SQLite's 256 MiB mmap window (`SQLITE_MMAP_SIZE_BYTES`) and existing B-tree indexes (`idx_security_rule_events_timestamp`, `idx_security_rule_events_event_id`, `idx_security_rule_events_rule_id`).

#### Diff

```diff
--- a/crates/capsem-logger/src/schema.rs
+++ b/crates/capsem-logger/src/schema.rs
@@ -7,7 +7,11 @@
use tracing::warn;

/// Tables that must never be mirrored into the DB-owned in-memory schema.
-const DISK_ONLY_TABLES: &[&str] = &["event_body_blobs"];
+const DISK_ONLY_TABLES: &[&str] = &[
+ "event_body_blobs",
+ "security_rule_events",
+ "security_decision_events",
+];

pub const READY_SCHEMA_COLUMNS: &[(&str, &[&str])] = &[
```

### Impact & Verification
* **Startup time**: Reduced from ~165 seconds to sub-second.
* **Service memory footprint**: Drops from 57+ GiB to ~50 MiB.
* **Enforcement speed**: Unaffected; live security policy decisions are evaluated in-memory using compiled CEL rules in `capsem-core` and do not read SQLite.
* **Query performance**: Read queries (`GET /vms/:id/security/events`, `capsem triage`) hit the indexed B-tree on disk in < 1–2 ms without loading the entire historical ledger into memory.

Contributor guide

Open the contributing guide

Research direction

Start in crates/capsem-logger/src/schema.rs, then read DbReader::open in crates/capsem-logger/src/reader.rs and the synchronization path in crates/capsem-logger/src/schema/memory_sync.rs. Confirm the two security event tables are excluded from the in-memory mirror while event queries still use the disk database, then run the capsem-logger tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.