Incremental reindex DETACH DELETE degrades to full relationship-table scan, pins CPU for hours
- Dominant language
- Rust
- Stars
- 88
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Found while investigating a month-long recurring \"infigraph gets stuck\" report. A live `infigraph-mcp --worker` process was found alive ~19 hours, one thread pinned at 100% CPU the entire time. Process sampling (`sample`) showed the hot thread stuck in:
```
lbug::storage::RelTable::detachDelete
→ detachDeleteForCSRRels
→ RelTableData::delete_
→ RelTableData::findMatchingRow
→ CSRNodeGroup::scan
→ CSRNodeGroup::scanCommittedPersistentWithoutCache
→ ChunkedNodeGroup::scan
→ VersionInfo::getSelVectorToScan / VectorVersionInfo::getSelVectorForScan
```
100% of sampled stack traces across multiple 1-5s samples landed in this call chain — not idle, not deadlocked, genuinely CPU-bound.
## Root cause
`delete_files_data()` (`crates/infigraph-core/src/graph/kuzu_backend.rs:548-577`) runs, on every incremental reindex (i.e. every file save while a watcher is active):
```cypher
MATCH (s:Symbol) WHERE s.file IN [...] DETACH DELETE s
```
`DETACH DELETE` requires Kuzu to find and remove every relationship touching each deleted `Symbol` node (`CALLS`, `CONTAINS`, `INHERITS`, `TESTED_BY`, etc.) before deleting the node. There's no index accelerating that relationship lookup -- `findMatchingRow` falls through to a linear scan (`CSRNodeGroup::scan`) over the whole rel table's committed node-groups.
As a project's graph accumulates edges over a long watch session, each incremental delete gets progressively more expensive. On a large enough graph (or long enough session), this scan can pin a CPU core for hours, which presents to the user as \"infigraph is stuck\" rather than an obvious crash.
## Reproduction context
Found on a real dev machine's infigraph-mcp worker for this repo (5478 symbols, 5421 call edges at time of discovery) after a long session. Not yet reproduced via a minimal synthetic repro -- next step is to build one (e.g. a large synthetic graph + repeated single-file incremental delete+reinsert) to confirm the scaling behavior and measure the actual growth curve.
## Relates to
- #46 (watcher holds graph write lock for whole lifetime) -- a slow `DETACH DELETE` makes that lock-holding problem strictly worse, since the lock is held for the duration of this scan.
- The deferred \"DB write serialization daemon\" idea.
## Proposed direction (not yet implemented)
Needs investigation into Kuzu's/lbug's actual capabilities before committing to an approach:
- Check whether Kuzu supports an index on relationship endpoint columns that would make `findMatchingRow` a lookup instead of a scan.
- Consider whether `delete_files_data` can avoid `DETACH DELETE` entirely -- e.g. delete outgoing/incoming edges explicitly via a targeted `MATCH ()-[r]->() WHERE ... DELETE r` first (if that has a faster path than the implicit detach), then delete the bare node.
- Consider batching/throttling incremental deletes rather than running them synchronously inline in the watcher's hot path.
Contributor guide
Research direction
Start in crates/infigraph-core/src/graph/kuzu_backend.rs:548-577 and trace delete_files_data() through the reported detachDelete call chain. Build the proposed large synthetic graph with repeated single-file incremental delete-and-reinsert operations, then inspect Kuzu index capabilities and measure whether the relationship scan grows. Done means a validated scaling result and an agreed, verified path that avoids the hours-long CPU-bound delete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100