galaxyproject / galaxyproject/loom

Galaxy tool runs are invisible to the poller, so the agent stalls until nudged

Open
#413 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
14
Forks
12
Avg merge
6d 5h
Merged PRs (30d)
17

Description

## Summary

An analysis driven with Galaxy **tool runs** (rather than workflow invocations) stalls after every submit, and only resumes when the user types something. The background poller cannot see tool runs at all, and the execution prompt tells the model to stop anyway — promising a watcher that, on this path, does not exist.

## Evidence

From one real session's `activity.jsonl` — idle time before the user had to prompt:

| Idle | What the user typed |
|---|---|
| 931.4 min | `status?` |
| 75.9 min | `what is the progress?` |
| 66.2 min | `where are we?` |
| 61.8 min | `I think bwa is done, no?` |
| 49.5 min | `status?` |
| 21.2 min | `why are we pausing?` |

27 user prompts against 119 tool calls. A large share of those prompts are pure nudges carrying no new instruction.

The same session's `notebook.md`:

```
3 ```loom-session blocks
0 loom-invocation blocks
0 in_progress markers
31 mentions of bwa / fastp
```

## Cause

**1. The poller only tracks workflow invocations.** `galaxy-poller.ts` ticks every 15s, but its first act is `hasInProgressInvocations()`, which scans `notebook.md` for `loom-invocation` blocks with `status: in_progress`. Those blocks are written by `galaxy_invocation_record`, which the model calls after invoking a **workflow**.

A Galaxy **tool** run (`galaxy_run_tool`) writes no such block. So on this session every tick took the cheap early-return path: no Galaxy call, no status advance, no completion toast. Nothing was watching bwa.

**2. The prompt tells the model to stop regardless.** `execution-commands.ts:44`:

> "hand control back to the user: say it's submitted and running in the background (the Activity tab shows live progress), and STOP. Do NOT sit in this turn polling the invocation to completion — **a background poller advances its status automatically and the user is notified when it finishes**."

That is true for workflow invocations and false for tool runs. The model is behaving correctly against an instruction whose premise doesn't hold, so the turn ends and nothing resumes it. The Activity tab likewise shows no progress, because nothing is advancing counters.

**3. Even when tracking works, completion does not resume the agent.** The poller's terminal-state handler toasts:

```
✅ Galaxy: "" finished (N jobs ok) — ask me to verify the outputs.
❌ Galaxy: "" failed (N job error(s)) — ask me to investigate.
```

The human is the messenger by design. So a tracked workflow still needs a nudge; the user is merely told when to give it.

## Proposed fixes

**A. Make tool runs pollable.** Give tool submissions the same treatment invocations get — a recorded, pollable notebook block — so the poller advances them and the existing toast/counter machinery lights up. `galaxyGetJobDetails()` (`galaxy-api.ts:136`) already returns job `state`, so the polling primitive exists; what is missing is the record at submit time and a poller branch that advances jobs.

**B. Wake the agent on completion instead of only toasting.** Deliver the completion as a follow-up to the brain so it verifies outputs and continues on its own, rather than emitting text that asks the user to relay. This is the fix that actually removes the nudging.

Two constraints worth stating up front:

- It must **queue**, not interrupt. Sending a prompt to a brain mid-turn fails with `Agent is already processing. Specify streamingBehavior ('steer' or 'followUp')`.
- It makes the agent run turns unattended. That is the point, but it changes the cost profile of a long analysis and should probably be opt-in.

**C. Make the "STOP, a poller has this" instruction conditional** on the work actually being tracked, so the prompt stops promising a watcher that isn't there. Worth doing regardless of A's timeline.

## Notes

- A is the highest value per unit of effort: poller, block format, counters, and toasts all exist; only the record and a poller branch are missing.
- B is the one users will feel, and is best built on A rather than instead of it.
- Related: #410 (Galaxy MCP timeouts), which is what made several of these tool calls slow enough to notice in the first place.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with galaxy-poller.ts and the Galaxy tool submission path, then compare the tracking blocks written by galaxy_invocation_record with galaxy_run_tool. Read galaxy-api.ts:136 for the existing job-state lookup and execution-commands.ts:44 for the conditional stop instruction. Done means tool runs are recorded and advanced by the poller, with completion behavior and prompt wording matching the selected tracking flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.