intuit / intuit/infigraph

MCP worker: panic hook panics recursively and aborts with multi-GB core dumps; separately OOMs at 110 GB RSS

Open
#65 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
88
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Summary

Over a single day of normal MCP use on one repo (~1.4M LOC Go/Kubernetes project), `infigraph-mcp` 3.2.16 hit three distinct failure modes: two OOM kills at ~110 GB RSS, six segfaults, and repeated aborts that each dumped a multi-gigabyte core file into the current working directory.

The aborts are the easiest to fix and they make everything else worse, so I'll lead with those.

## 1. The panic hook panics, recursively, and turns every panic into an abort

`install_panic_hook` logs through `mcp_log`, which uses `_eprint`. When the write to stderr fails - which it does the moment the MCP pipe to the agent is gone - `_eprint` panics with `failed printing to stderr`. That panic re-enters the hook, which calls `mcp_log` again, which panics again:

```
#3 std::sys::pal::unix::abort_internal
#4 std::process::abort
#5 std::panicking::panic_with_hook
...
#10 std::io::stdio::_eprint
#11 infigraph_mcp::mcp_log
#12 infigraph_mcp::install_panic_hook::{closure#0}
#13 std::panicking::panic_with_hook
...
#18 std::io::stdio::_eprint
#19 infigraph_mcp::mcp_log
```

It only stops because Rust's panicked-while-panicking guard calls `abort()`.

Two consequences:

- **Any** recoverable panic in a worker becomes a hard abort.
- Because the process aborts rather than exits, it dumps its entire heap. The two cores this left in my repo are **1.7 GB and 1.1 GB**.

It also explains why `~/.infigraph/mcp.log` contains no error lines at all across all of these crashes - the logging call is the thing that dies, so nothing is ever recorded. From the outside it just looks like the server vanished.

Suggested fix: the panic hook must not use `eprintln!`/`_eprint`. Write to a raw `File`/`Stderr` handle and discard the result, or route it to the log file with the error swallowed. A panic hook that can itself panic is never safe.

## 2. OOM: 8.7 TB virtual, 110 GB resident

Twice today, killed by the kernel OOM killer:

```
Aug 21 17:35:19 kernel: Out of memory: Killed process 490216 (infigraph-mcp)
total-vm:8700240216kB, anon-rss:105359944kB, file-rss:2772kB, pgtables:542476kB

Aug 21 18:43:10 kernel: Out of memory: Killed process 695487 (infigraph-mcp)
total-vm:8700240216kB, anon-rss:110447384kB, file-rss:3000kB, pgtables:537648kB
```

That is 8.7 TB of address space and ~110 GB resident in a single process, and the identical `total-vm` in both suggests one specific allocation path rather than gradual growth.

This is related to #46 but not the same thing. #46 describes orphaned watchers pinning ~100-425 MB each and accumulating; I see that too (below), but no amount of accumulation puts 8.7 TB of VA into one process.

## 3. Segfaults

Six today, all with the identical signature - a write to a near-null pointer, on a worker thread, inside libc:

```
Aug 21 13:56:43 kernel: infigraph-mcp[3931683]: segfault at 40 ip 00007f2d775b81e2 sp 00007713685fcd78 error 6 in libc.so.6[1b81e2,...]
Aug 21 15:58:25 kernel: infigraph-mcp[139315]: segfault at 40 ip 00007f84eb7b81e2 sp 00007f84dd3fcf38 error 6 in libc.so.6[1b81e2,...]
Aug 21 17:29:48 kernel: infigraph-mcp[486288]: segfault at 40 ip 00007fc6ddfb81e2 sp 00007fc6d1ffcf38 error 6
Aug 21 17:30:39 kernel: infigraph-mcp[488537]: segfault at 40 ip 00007f08071b81e2 sp 00007f07f1ffcf38 error 6 in libc.so.6[1b81e2,...]
Aug 21 18:36:06 kernel: infigraph-mcp[688871]: segfault at 40 ip 00007f6c95db81e2 sp 00007f6c91bfbf38 error 6 in libc.so.6[1b81e2,...]
Aug 21 18:38:08 kernel: infigraph-mcp[693914]: segfault at 40 ip 00007f2f56db81e2 sp 00007f2f35ffcf38 error 6 in libc.so.6[1b81e2,...]
```

`error 6` is a user-mode write to a not-present page, faulting address `0x40`, always at the same offset into libc. Given the OOM behaviour above, an unchecked allocation result is a plausible common cause, but I haven't confirmed that.

## 4. Worker processes accumulate and never exit

12 live `infigraph-mcp` processes at the time of writing, the oldest 1h32m old:

```
PID RSS VSZ ELAPSED COMMAND
533030 258336 3432384 01:32:42 infigraph-mcp
739445 61344 3331740 22:12 infigraph-mcp
761647 16852 3297968 13:53 infigraph-mcp
739349 16848 3297968 22:12 infigraph-mcp
721806 16688 3297968 27:03 infigraph-mcp
... 7 more
```

`~/.infigraph/mcp.log` shows a fresh server starting every ~500s, each one losing the lock race:

```
[...] WARN: Another MCP instance holds the lock — running without watchers
[...] INFO: MCP server started
[...] INFO: stdin loop exited
```

`stdin loop exited` is logged but the process stays resident. This looks like the same shape as #9, which was closed as macOS-specific - this is Linux.

## 5. Core dumps land in the user's working directory

Not an infigraph bug as such, but worth knowing: on a system where `kernel.core_pattern` is a relative path, the aborts in section 1 drop gigabyte-scale cores into whatever directory the worker happened to be in, which is the user's repo. I had 2.8 GB of `core.` files sitting next to my source. Fixing section 1 removes the abort, and with it these.

## Environment

- infigraph 3.2.16 (mise-installed binary), MCP server via Claude Code
- CachyOS Linux, kernel 7.1.8-1-cachyos, x86_64
- Repo under analysis: kserve/kserve (Go, ~1.4M LOC), indexed with embeddings (`embeddings.bin` 111 MB, `graph` 91 MB)

Contributor guide

Open the contributing guide

Research direction

Start with install_panic_hook, mcp_log, and _eprint, then reproduce the failed-stderr path described in the report. Separately inspect the MCP worker's stdin loop and the startup/lock handling associated with “stdin loop exited” and accumulating processes. Done requires isolating the failure modes and adding verified fixes or tests for the confirmed causes.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
backend, devtools, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.