intuit / intuit/infigraph

Incremental reindex DETACH DELETE degrades to full relationship-table scan, pins CPU for hours

Open
#51 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
88
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Summary

Found while investigating a month-long recurring \"infigraph gets stuck\" report. A live `infigraph-mcp --worker` process was found alive ~19 hours, one thread pinned at 100% CPU the entire time. Process sampling (`sample`) showed the hot thread stuck in:

```
lbug::storage::RelTable::detachDelete
→ detachDeleteForCSRRels
→ RelTableData::delete_
→ RelTableData::findMatchingRow
→ CSRNodeGroup::scan
→ CSRNodeGroup::scanCommittedPersistentWithoutCache
→ ChunkedNodeGroup::scan
→ VersionInfo::getSelVectorToScan / VectorVersionInfo::getSelVectorForScan
```

100% of sampled stack traces across multiple 1-5s samples landed in this call chain — not idle, not deadlocked, genuinely CPU-bound.

## Root cause

`delete_files_data()` (`crates/infigraph-core/src/graph/kuzu_backend.rs:548-577`) runs, on every incremental reindex (i.e. every file save while a watcher is active):

```cypher
MATCH (s:Symbol) WHERE s.file IN [...] DETACH DELETE s
```

`DETACH DELETE` requires Kuzu to find and remove every relationship touching each deleted `Symbol` node (`CALLS`, `CONTAINS`, `INHERITS`, `TESTED_BY`, etc.) before deleting the node. There's no index accelerating that relationship lookup -- `findMatchingRow` falls through to a linear scan (`CSRNodeGroup::scan`) over the whole rel table's committed node-groups.

As a project's graph accumulates edges over a long watch session, each incremental delete gets progressively more expensive. On a large enough graph (or long enough session), this scan can pin a CPU core for hours, which presents to the user as \"infigraph is stuck\" rather than an obvious crash.

## Reproduction context

Found on a real dev machine's infigraph-mcp worker for this repo (5478 symbols, 5421 call edges at time of discovery) after a long session. Not yet reproduced via a minimal synthetic repro -- next step is to build one (e.g. a large synthetic graph + repeated single-file incremental delete+reinsert) to confirm the scaling behavior and measure the actual growth curve.

## Relates to

- #46 (watcher holds graph write lock for whole lifetime) -- a slow `DETACH DELETE` makes that lock-holding problem strictly worse, since the lock is held for the duration of this scan.
- The deferred \"DB write serialization daemon\" idea.

## Proposed direction (not yet implemented)

Needs investigation into Kuzu's/lbug's actual capabilities before committing to an approach:
- Check whether Kuzu supports an index on relationship endpoint columns that would make `findMatchingRow` a lookup instead of a scan.
- Consider whether `delete_files_data` can avoid `DETACH DELETE` entirely -- e.g. delete outgoing/incoming edges explicitly via a targeted `MATCH ()-[r]->() WHERE ... DELETE r` first (if that has a faster path than the implicit detach), then delete the bare node.
- Consider batching/throttling incremental deletes rather than running them synchronously inline in the watcher's hot path.

Contributor guide

Open the contributing guide

Research direction

Start in crates/infigraph-core/src/graph/kuzu_backend.rs:548-577 and trace delete_files_data() through the reported detachDelete call chain. Build the proposed large synthetic graph with repeated single-file incremental delete-and-reinsert operations, then inspect Kuzu index capabilities and measure whether the relationship scan grows. Done means a validated scaling result and an agreed, verified path that avoids the hours-long CPU-bound delete.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.