galaxyproject / galaxyproject/loom

Right-size always-on system prompt + tool surface for smaller open models

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
14
Forks
12
Avg merge
6d 5h
Merged PRs (30d)
17

Description

## Symptom

Running Loom in non-interactive mode (`-p`) against gpt-oss-120b (via a local litellm relay) consistently hangs on the first turn -- the model produces only a `reasoning_content` block and never reaches a `content` or `tool_use` response. The session ends with a clean `notebook_snapshot` and zero work done.

Same model + same prompt + a tiny ad-hoc tool surface (single tool, no system prompt extras) returns clean tool calls in <1s. Same model on a focused Loom prompt with `--no-skills --no-context-files --thinking off` works end-to-end (real tool calls, real BRC MCP queries, notebook written).

So the model is capable. The failure mode is the *combined weight* of Loom's always-on context: skill-router blurb in `extensions/loom/context.ts:561-609`, MCP tool lists (galaxy-mcp's ~20 + brc-analytics' 12 + pi built-ins + web access + Loom extension tools), plus AGENTS.md / CLAUDE.md ambient injection. At full surface, smaller open models exhaust themselves on reasoning before producing output.

## Reproducer

```bash
mkdir /tmp/loom-tier0-repro && cd /tmp/loom-tier0-repro
LOOM_FRESH_SESSION=1 node /path/to/loom/bin/loom.js -p \
--provider litellm --model gpt-oss-120b \
"Use the brc_analytics_search_organisms tool to find Plasmodium falciparum and report the result."
# Hangs on first turn. Activity log shows assistant message with only 'thinking' part, no text/tool_use.

# Same prompt with reduced surface succeeds:
LOOM_FRESH_SESSION=1 node /path/to/loom/bin/loom.js -p \
--provider litellm --model gpt-oss-120b \
--no-skills --no-context-files --thinking off \
"Use the brc_analytics_search_organisms tool to find Plasmodium falciparum and report the result."
# Calls the tool, gets NCBI taxonomy 5833 + 27 assemblies, writes a markdown table response.
```

This isn't gpt-oss-120b-specific -- it's a class effect. Llama-4-Maverick / Sonnet / Opus would likely shrug it off. But "smaller open models hit the wall sooner" is exactly the audience Loom is trying to support for air-gapped / cost-sensitive / Galaxy-community deployments.

## Proposed approach

Treat always-on context as a **budget**, not a bag of features that grow as we ship. Three concrete steps:

1. **Audit current Tier-0 size.** Add a one-time startup diagnostic that prints the assembled system prompt + tool-list size in tokens (per provider's tokenizer where possible, char-count approximation otherwise). Logs to stderr behind `--verbose`, doesn't change UX. Lets us measure.

2. **Define a Tier-0 budget per model class.** Probably ~5K tokens for "small-open" models, ~15K for hosted frontier. Stored as model metadata in `~/.pi/agent/models.json` or shipped in Loom's defaults. When the assembled Tier-0 exceeds the configured budget, log a warning at startup ("system prompt 12.3K tokens; configured budget 5K -- consider reducing skills router or other always-on sections").

3. **Make `--print` mode default to a leaner Tier-0.** In non-interactive runs the user almost never benefits from the welcome greeting, the skill router (they're invoking a single shot, not browsing skills), or ambient CLAUDE.md context. Today they have to pass three flags. Reasonable defaults for `--print`:
- skip the session-start welcome (already done if `LOOM_FRESH_SESSION=1`)
- skip skill-router prose (skill *fetching* still works on demand)
- keep CLAUDE.md / AGENTS.md (low-cost, often useful)

Make the existing flags overrides rather than required for bare survival.

## Why this matters now

Loom's always-on context grows naturally as features ship -- new typed notebook blocks add summary lines to the digest, new MCP registrations add tool descriptions, new grounding sources add router blurbs. Each addition individually passes the "+1 line is fine" test; the aggregate slowly excludes smaller open models. Catching the budget question now is cheap; catching it after several more features compound is a refactor.

This is also the empirical motivation for context-management discipline as Loom rides on Galaxy's full surface area (~8K tools, IWC workflows, GTN, BRC organism / assembly / workflow catalog, etc.). The substrate is what makes Loom interesting; without budgeting the surface that actually hits the model, the substrate becomes the failure mode.

## Open questions

- Does pi expose a hook to introspect the assembled system prompt? If not, the audit diagnostic might need a cooperative addition upstream.
- For the `--print` defaults change: should the lean profile be opt-out (`--print` defaults to lean, `--print --full` to keep current behavior), or opt-in via a separate flag (`--print-lean`)? Opt-out is safer if pre-existing scripts rely on the current behavior; opt-in is more discoverable.
- Is there a clean way to express "fetch this skill on demand but don't put its router blurb in Tier-0" -- so the skill is reachable via `skills_fetch` without paying the prompt cost? That's the architectural question behind this whole thread.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.