galaxyproject / galaxyproject/loom

Run multiple analyses at once: a herd view, and the per-session state it needs

Open
#417 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
14
Forks
12
Avg merge
6d 5h
Merged PRs (30d)
17

Description

## The need

A user running several analyses at once — different datasets, different plans, sometimes different Galaxy servers — has no supported way to do it today. Worth treating as a real workflow rather than an edge case: Galaxy work is long-running and mostly waiting, which is exactly when someone starts a second thing.

## Where we are today

**Two Orbit processes are blocked by design.** `app.requestSingleInstanceLock()` (`app/src/main/main.ts:39`) makes a second launch quit and forward its argv to the running instance via `second-instance`. A separate `--user-data-dir` does give the second instance its own lock — verified working, two Orbits side by side — but that is a workaround, not a feature.

**Same-directory concurrency is unsafe.** `withNotebookLock` (`extensions/loom/notebook-writer.ts:35`) is a per-path promise chain in a `Map` in **one process's memory**, with an explicit comment that it avoids "paying for an OS-level lock". It cannot see another process. Two brains on one `notebook.md` race read-modify-write and silently lose updates — the exact failure the lock exists to prevent, one level up. Both would also run their own 15s Galaxy poller over the same invocations and both rewrite the `session.jsonl` symlink.

**This argues for multiple sessions in ONE Orbit, not multiple Orbits.** In-process, the existing lock becomes correct rather than being worked around, and the single-instance lock stops being an obstacle. It is also the right split per `CLAUDE.md`: user-driven parallelism (a human opens and steers each analysis) is shell work, distinct from the model-driven delegation discussed in #407.

## Scope: per-session Galaxy connection

This is the load-bearing piece. Without it, two windows quietly share one Galaxy connection — which looks fine until someone switches servers.

### Smaller than it first appears

A naive reading suggests threading a config object through the 67 `GALAXY_URL` / `GALAXY_API_KEY` references across 19 files. Not needed:

- **Each analysis already runs its own brain process.** Orbit spawns `node bin/loom.js --mode rpc` per `AgentManager`.
- **Orbit already builds the brain env per spawn** — `buildBrainEnv(fresh)` at `app/src/main/agent.ts:393`, and `buildSecretEnv()` re-reads config on every spawn.
- So credentials travelling in **process env are already per-session-capable**. The 18 direct `process.env.GALAXY_*` reads inside `extensions/loom/` are fine as they stand, because `process.env` *is* the session boundary. `getGalaxyConfig()` (`galaxy-api.ts:54`) stays the brain-side chokepoint for normalization.

The problem is not the transport. It is that the values are **derived from global state** — one active profile, one `mcp.json`.

### 1. Per-session `mcp.json` — the actual blocker

`bin/loom.js:309` writes `join(agentDir, "mcp.json")`, and `agentDir` is `process.env.PI_CODING_AGENT_DIR || ~/.pi/agent` (`bin/loom.js:157`). Every brain launch rewrites that one file with a whole-object assignment, so two analyses starting close together race, and the later launch's Galaxy server wins for both.

`PI_CODING_AGENT_DIR` looks like the seam — and pi-mcp-adapter honours it too (`agent-dir.ts:4`) — but it relocates the **whole agent dir**, which also holds `auth.json` and `models.json`. Pointing each session at its own dir would fragment LLM auth: every analysis would need its own OAuth login. That is worse than the problem.

Three options:

| | Approach | Cost | Risk |
|---|---|---|---|
| a | Per-session `PI_CODING_AGENT_DIR`, symlink `auth.json`/`models.json` back to the shared dir | small | symlink hygiene; refresh writes go through the link |
| b | Add a narrower `mcp.json` path override, upstream in pi-mcp-adapter (and pi) | small locally, needs upstream | release coupling |
| c | Keep one `mcp.json`, make the write atomic and merge-only | smallest | does not solve *different servers per session*, only the race |

(c) is worth doing regardless — the current whole-object assignment is a lost-update hazard on its own. But only (a) or (b) actually supports two analyses on different Galaxy servers.

### 2. Session → profile binding

`switchProfile()` (`extensions/loom/profiles.ts:240`) writes the shared profile store **and** mutates `process.env`. `/connect` in one window therefore repoints the stored active profile for every future brain start. Needs a per-session notion of "this analysis uses profile X", with `/connect` scoped to the current session rather than global.

### 3. Shell work — the "herd" view

`main.ts:287` holds a single `agentManager`. Multiple sessions means several `AgentManager`s, each with its own cwd, brain process, and lifecycle, plus UI to move between them. This is the bulk of the effort and it is ordinary shell work.

The UI shape worth aiming at is a persistent sidebar listing every live analysis — the pattern Claude Code itself uses for spaces/agents:

```
analyses ● cyclospora running
✓ mt-dev-panel 2 outputs to verify
✓ brc-assembly done
! rnaseq-pilot 1 job failed
```

Each row is an analysis directory, with its Galaxy profile as the sub-line (the way that sidebar shows a branch under each repo), a status glyph, and an attention count. Selecting a row swaps the main pane.

The important part is that this is **not just navigation** — it is where the polling work pays off:

- Status per row comes straight from what the poller already maintains: `loom-invocation` and (after #414) `loom-job` blocks carry `in_progress` / `completed` / `failed`, plus job counters.
- The completion toast becomes a **persistent per-session badge**. Today a toast fires once and is gone; if you were in another window, you missed it. A badge is exactly what "which analysis needs me?" wants, and it does not depend on the user being present at the moment a run finishes.
- With auto-resume (#416) a row can advance itself — "finished → verified → waiting for you" — so the badge means *decision needed*, not *chore needed*.

That combination is the actual answer to "the agent constantly needs nudging" (#413): several analyses progressing on their own, and one place that shows which one wants a human.

It also constrains the backend: per-session status must be queryable by the shell without switching to that session, so the notebook blocks (or a summary derived from them) need to be readable per analysis directory, not just for the active one.

### 4. Guard rail (do this first, cheaply)

Whatever else happens, refuse to open an analysis directory that another session already holds — an OS-level lock file in the cwd, or a check at open time. The in-process notebook lock cannot protect across processes, and today nothing stops a user pointing two Orbits at one directory and quietly corrupting the notebook.

## Suggested order

1. **Guard rail (4)** and **atomic `mcp.json` write (1c)** — small, independently valuable, reduce the blast radius of what people can already do by hand today.
2. **Per-session Galaxy connection (1a or 1b + 2)** — the load-bearing change.
3. **Multi-session shell (3)** — the visible feature, once the state underneath is genuinely per-session.

Doing (3) first would produce two windows sharing one Galaxy connection and one `mcp.json`, which looks like it works right up until someone switches servers.

## Notes

- Verified today: separate `--user-data-dir` + separate `LOOM_CWD` runs two Orbits without collisions. That is the current best workaround and worth documenting even before any of the above ships.
- `auth.json` sharing is fine as-is — pi does its own locking around OAuth refresh.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading app/src/main/main.ts, app/src/main/agent.ts, bin/loom.js, extensions/loom/profiles.ts, and extensions/loom/notebook-writer.ts, then trace how sessions create agent directories and write mcp.json. Implement the guard rail and session isolation before the multi-session shell. Done means concurrent analyses keep separate Galaxy profiles and state, and the UI can show and select each session with persistent status or attention indicators.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
desktop, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.