mitodl / mitodl/agent-kit

witan inject-context: cold graph read takes ~20s, exceeding the 15s hook timeout it installs

Open
#349 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3
Forks
0
Avg merge
18h 58m
Merged PRs (30d)
81

Description

Expected Behavior

witan inject-context completes inside the 15s UserPromptSubmit timeout that witan setup installs, and the rendered workflow-context block reaches the agent. The source's own cost model assumes this — each omnigraph read is budgeted at "~1-2s on a large graph":

https://github.com/mitodl/agent-kit/blob/e9a32dabc49ff4daf4f6cd27e319c99ae4c1a3aa/mcp/servers/witan/witan/context.py#L33

Current Behavior

Against the deployed server (https://witan.ol.mit.edu/mcp), a cold witan inject-context — any prompt where the 30s output cache has expired — takes 16–23s, so Claude Code kills the hook and throws away its output:

UserPromptSubmit hook timed out after 15s — output discarded. Raise the hook's "timeout" to allow more time.

The net effect is the worst of both: the user waits the full 15s and gets nothing. Apart from that one line the failure is silent, and the agent proceeds with no knowledge of active projects, in-flight branch tasks, or claimed by markers — the exact duplicate-work hazard the hook exists to prevent.

Measured on macOS 15.6, ol-agent-kit 0.1.3, against mitodl/ol-data-platform on main. Each run was taken after letting the 30s output cache expire, so each is a true cold path:

run witan inject-context witan code inject-context (for contrast)
1 19.40s 1.25s
2 15.83s 0.85s
3 21.14s 0.96s
4 23.22s 0.80s
5 18.96s 0.98s
6 22.58s 0.78s
7 10.38s

Mean 18.8s, median ~19.4s, min 10.38s, max 23.22s — 6 of 7 cold runs exceed the 15s timeout. The one that came in under it still took 10.4s, i.e. 5–10× the budgeted cost. So this isn't pure tail latency (the median sits well above the limit), but it is variable enough that the failure is intermittent rather than constant — which is how it presents: "I keep hitting this error," not "it always fails." Warm (cache-hit) runs are 0.6–0.9s.

For scale, the graph being read has 19 active WorkflowProjects, 184 ready tasks, and 11 tasks linked to the current branch; the rendered block is ~5.3KB. witan code inject-context reads a local index and is never near the limit — this is specific to the remote workflow read.

Steps to Reproduce
  1. witan setup to install the hooks — it writes timeout_seconds=15:
    https://github.com/mitodl/agent-kit/blob/e9a32dabc49ff4daf4f6cd27e319c99ae4c1a3aa/mcp/servers/witan/witan/setup.py#L68-L77
  2. Configure a target pointing at the deployed service (remote_url = "https://witan.ol.mit.edu/mcp") with a graph of comparable size.
  3. From a git repo covered by that target, time a cold run:
    echo '{"prompt":"t","cwd":"'"$PWD"'"}' | /usr/bin/time -p witan inject-context >/dev/null
    
  4. Wait more than 30s so the output cache expires, then repeat. Every cold run exceeds 15s.

witan inject-context --debug confirms which path was taken — served from output cache on the fast runs, and remote_url=... output_cache_ttl=30.0s on the slow ones.

Possible Solution

Roughly in order of how durable each is:

  1. Reduce the read cost — the actual defect. Issuing several full store scans per prompt against a remote deployment is what doesn't fit the budget. If the scans can't be made cheap, the hook could fetch a server-side pre-rendered block in a single round trip rather than several reads it then renders locally.
  2. Raise timeout_seconds in both installers above the observed distribution. Worth noting the pi extension is tighter still at timeout: 5000, so it presumably never lands a block on a graph this size either:
    https://github.com/mitodl/agent-kit/blob/e9a32dabc49ff4daf4f6cd27e319c99ae4c1a3aa/configs/pi/extensions/codegraph.ts#L102-L104
  3. Reconsider the trigger. inject-context's output depends only on graph_uri|repo|branch and never reads the prompt text — see the cache key:
    https://github.com/mitodl/agent-kit/blob/e9a32dabc49ff4daf4f6cd27e319c99ae4c1a3aa/mcp/servers/witan/witan/context.py#L89-L92
    That makes it session/repo state on a per-prompt trigger. SessionStart would pay the cost once per session, where a pause is unremarkable. The counter-argument is that per-prompt re-injection is what re-delivers the block after mid-session compaction drops it — so this is a real trade, not an obvious win.
  4. Raise the default _OUTPUT_CACHE_TTL (currently 30.0s):
    https://github.com/mitodl/agent-kit/blob/e9a32dabc49ff4daf4f6cd27e319c99ae4c1a3aa/mcp/servers/witan/witan/context.py#L40
    At 30s, ordinary think-and-type time between prompts lands on the cold path constantly. The cache file lives in the shared TMPDIR keyed by graph/repo/branch, so a longer TTL benefits every concurrent session at the cost of staler claimed by markers.

Local workaround applied in the meantime: timeout raised to 45 and WITAN_CONTEXT_TTL=300 set inline on the hook command.

Additional Details

Possibly related, unverified: witan session-checkpoint (the Stop hook) is installed with the same timeout_seconds=15 and reaches the same deployment through _srv() to call workflow_session_end:

https://github.com/mitodl/agent-kit/blob/e9a32dabc49ff4daf4f6cd27e319c99ae4c1a3aa/mcp/servers/witan/witan/setup.py#L74-L77

If that write also crosses 15s it would be killed the same way, leaving sessions open and their handoff summaries unwritten. That would match a symptom visible in the graph itself, where injected context reports sessions closed by the sweeper: "this session was left open for more than 6h and recorded no handoff summary of its own. It was not checkpointed, so nothing about what it did is known." I have not measured the checkpoint path, so I'm flagging the hypothesis rather than asserting it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mcp/servers/witan/witan/context.py, especially the cache key, output cache TTL, and remote-read path, then inspect setup.py for the installed hook timeout and configs/pi/extensions/codegraph.ts for the tighter timeout. Reproduce a cold witan inject-context run after the cache expires and compare it with the local code path. Done means the cold workflow-context injection reliably completes within its configured hook budget without discarding output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.