stablyai / stablyai/orca

[Feature]: Decouple workspace filesystem from outer git worktree — nested multi-repo / governance-root layouts

Open
#8,702 0 comments 7 reactions 1 assignee Claimed by @brennanb2025 View on GitHub
enhancement
Dominant language
TypeScript
Stars
72.1k
Forks
4.7k
Avg merge
14h 54m
Merged PRs (30d)
520

Description

## Summary

Orca’s current model effectively equates:

> **Project / Workspace ≈ one git repository root + isolated `git worktree add`**

That works extremely well for single-repo parallel agents. It breaks for a common professional layout:

> **Outer governance / meta repository + nested (or sibling) product repositories**, where AI agents need the **whole directory tree as context**, but git isolation must happen **per product repo**, not on the outer root.

This is not an edge case of “messy repos.” It is a deliberate architecture used for AI-assisted multi-service development. Today Orca forces a false choice between **correct product isolation** and **complete agent context**. That is a product design limitation relative to editors like Zed / VS Code / IntelliJ that already treat **filesystem workspace** and **git roots** as separate concepts.

Related (please keep this issue focused on the *model* mismatch, not only UI chrome):

- #7900 — VS Code-style multi-root workspaces
- #1099 — multi-repo diffs in one workspace
- #7568 — multi-repo project groups with PR-aware workspaces
- #2654 — workspaces decoupled from new worktrees
- #8675 — multi-root git graph
- #1602 — packages / grouping multiple repositories
- #5379 — multi-repo workflow questions
- #6357 — monorepo worktree file visibility bugs

This issue is specifically about **nested multi-git under a meta root** (including outer-git + ignored inner checkouts), and why **worktree-from-outer-root** is the wrong primitive for AI agent workspaces.

---

## Real-world layout that breaks Orca (reproducible shape)

```text
CliProxy/ ← outer git repo (governance / meta)
├── .git/
├── AGENTS.md ← agent operating rules for the whole product surface
├── docs/ ← plans, RCAs, reviews, issue writeups
├── rules/ ← quality / evidence standards
├── scripts/ ← doc validators, cross-repo helpers
├── paseo.json ← local agent orchestration config
├── .gitignore ← explicitly ignores product checkouts

├── CliRelay/ ← independent product git repo (backend)
│ └── .git/
└── codeProxy/ ← independent product git repo (frontend)
└── .git/
```

Important details:

1. **Outer repo is real git**, not just a folder. It owns process docs, agent instructions, review artifacts, and cross-repo conventions.
2. **Inner repos are real independent remotes** with their own `dev`/`main`, CI, PR, and release pipelines.
3. Outer `.gitignore` often contains:

```gitignore
# product repos managed by their own .git
CliRelay/
codeProxy/
```

4. Implementation policy (common in agent-heavy teams):
- Read context from the **workspace root** (outer tree).
- Never implement product changes in the main checkout of `CliRelay` / `codeProxy`.
- Create **per-product** worktrees/branches/PRs.
- Cross-service tasks require **two product worktrees**, not one outer worktree.

This is intentionally **not** a classical monorepo (one git history for all code). It is also **not** only “unrelated folders.” It is a **meta-repo + product repos** topology.

---

## What happens in Orca today

### Path A — Open outer root as Project (`CliProxy`)

User creates a new workspace / feature worktree from the outer project.

**Observed result:**

- Orca creates an isolated worktree of the **outer** git repo.
- The new workspace contains outer tracked files: `AGENTS.md`, `docs/`, `rules/`, `scripts/`, etc.
- Product directories (`CliRelay/`, `codeProxy/`) are missing or empty / incomplete in the worktree view, because they are not part of the outer git tree (ignored independent checkouts).

**Screenshot-class failure mode:**

- Left: new Orca workspace looks “successful”
- Right file tree: only governance docs — no product source
- User annotation: “my full project structure is like this; Orca only got the outer layer”

### Path B — Open a product repo as Project (`CliRelay` or `codeProxy`)

Worktree isolation becomes correct for that product repo.

**But agent context is crippled:**

- Outer `AGENTS.md` / process rules disappear
- Cross-repo docs (`docs/plan`, `docs/review`) disappear
- Sibling product repo is no longer in the same workspace
- Agents re-learn conventions from partial context and invent conflicting workflows

### Forced false choice

| Open as root | Agent sees full context | Product code worktree correct |
|---|---|---|
| Outer meta repo | Yes | **No** |
| Inner product repo | **No** | Yes |

For AI-assisted development, both columns must be **Yes**. Orca currently makes them mutually exclusive.

---

## Why this is a product design issue (not user error)

### 1. Orca collapses two different concepts into one

Mature tooling separates:

| Concept | Meaning | Examples |
|---|---|---|
| **Workspace / project filesystem** | What humans and agents can see, search, and reason over | VS Code multi-root, Zed project roots, IntelliJ project |
| **Git repository root(s)** | Where history, branch, worktree, PR, and CI live | one or many git dirs under the workspace |

Orca’s agent isolation model currently binds:

```text
new workspace ≈ git worktree of the project’s primary git root
```

That is excellent for:

- one app repo
- parallel experiments on the same history
- “fan out 5 agents, each worktree, pick winner”

It is the wrong default for:

- meta-repo + product repos
- polyrepo product surfaces that still need one cognitive workspace
- AI workflows where **instructions live outside product remotes**

### 2. Nested / multi git is already first-class elsewhere

Editors and CLIs already handle multi-root git discovery:

- **Zed**: discovers multiple git repositories under a project and scopes SCM per root.
- **VS Code**: multi-root workspaces (`.code-workspace`) + multi-folder SCM.
- **IntelliJ / JetBrains**: multiple VCS roots, unified or filtered log, per-root operations.
- **Git itself**: nested repositories are normal; outer ignore of inner checkouts is normal; worktrees are per-repo.

Orca does not need to invent multi-git from scratch. It needs to stop assuming the **outermost git root is the only isolation boundary that matters**.

### 3. AI agents amplify the damage

For a human IDE user, missing a folder is annoying.

For an agent fleet, missing context is systemic:

- Agents do not know mandatory worktree policy in outer `AGENTS.md`
- Agents implement in the wrong place
- Agents skip required CI scripts documented only at meta root
- Agents cannot correlate frontend/backend contract changes with the plan/review docs that justify them
- Parallel agents become **context-isolated**, not just **git-isolated**

Orca’s value prop is parallel agents. Parallelism without shared project context produces expensive thrash.

### 4. “Just open the product repo” is not an acceptable workaround

Suggested workaround: always set Project root to `CliRelay` or `codeProxy`.

Why teams reject this:

1. **Information isolation** — outer agent rules and docs are load-bearing, not optional README fluff.
2. **Cross-service tasks** — one feature often needs backend + frontend + docs review artifact together.
3. **Process enforcement** — meta repo is where PR/merge/deploy policy is written for agents.
4. **Cognitive model** — users already think of `CliProxy/` as “the project,” because that is the folder they open every day.
5. **Tooling consistency** — other tools open the outer folder and still see inner repos.

So the workaround asks users to break their existing AI workflow to match Orca’s single-root assumption.

---

## Why this development structure exists (and why it is advantageous)

This is not accidental nesting. It is an intentional split of concerns optimized for agent-assisted shipping.

### Advantage 1 — Separation of **governance** from **product history**

Outer repo owns:

- agent instructions (`AGENTS.md`, rules)
- investigation plans / RCA docs
- review scorecards
- cross-repo conventions
- non-runtime scripts

Inner repos own:

- shippable code
- CI gates
- release tags
- deployment artifacts

Benefits:

- Product remotes stay clean for open-source or multi-team consumers.
- Process docs can evolve without polluting product commit history.
- Different access control: docs/process can be private while one product repo is public (or vice versa).

### Advantage 2 — Independent release trains with shared cognition

Backend and frontend often:

- ship on different cadences
- have different CI cost
- protect different branches
- need independent rollbacks

But agents still need one place to understand the whole system.

Meta-root + multi-product checkouts gives:

- independent git lifecycles
- one workspace brain for humans/agents

Classical monorepo gives shared cognition but couples release/history. Classical pure polyrepo gives independent release but fragments agent context. Meta-root layout is a third path.

### Advantage 3 — Hard isolation for implementation, soft sharing for context

A strong agent workflow often mandates:

- read everywhere relevant
- write only in task-specific product worktrees
- never dirty main product checkouts
- one PR per product repo

That requires **two layers**:

1. Shared read surface (outer tree + all product checkouts)
2. Per-repo write isolation (worktree/branch/PR)

Orca currently implements (2) by destroying (1) when the outer root is the project.

### Advantage 4 — Better AI maintainability over time

Putting agent policy in the outer repo creates a stable “operating system” for all coding agents (OpenCode, Claude Code, Codex, etc.):

- worktree rules
- required CI before PR
- deploy pipeline mental model
- log investigation playbooks
- “do not patch symptoms; fix model/root cause” standards

If each product repo only has partial instructions, agents diverge. Meta-root is the single source of agent truth.

### Advantage 5 — Documentation is a first-class artifact, not a side folder

In this structure, `docs/plan` and `docs/review` are not decorative. They are the handoff format between investigation agents and implementation agents.

If Orca worktrees drop those docs because they live outside the product git root, the investigation → implementation loop breaks.

### Advantage 6 — Matches how multi-agent orchestration already works outside Orca

Other local-first agent runners often accept:

- workspace cwd = outer folder
- task-level cwd / worktree = specific product repo

That maps cleanly onto human mental models. Orca’s “new workspace always = outer worktree” fights that model.

---

## Desired product model

Please treat these as separable primitives:

### A. Workspace (filesystem / agent context root)

A workspace is a **directory tree** the agent may read (and optionally write under policy).

It may contain:

- 0..N git repositories (nested or multi-root)
- non-git folders
- docs/rules that are not in the same remote as product code

### B. Git targets (operation roots)

Each discovered `.git` is a **git target** with its own:

- branch
- worktree set
- diff/SCM
- PR association
- agent write sandbox

### C. Agent runtime binding

When spawning an agent task:

- `context_roots = [workspace roots...]` (what the model can see)
- `write_root = selected git target worktree` (where isolation happens)
- optional additional read-only mounts (sibling repos, docs)

### Critical invariant

```text
Creating an isolated agent workspace must NOT require
rewriting the user’s project filesystem to a single outer worktree
when product code lives in nested/sibling git repositories.
```

---

## Concrete feature requests (prioritized)

### P0 — Nested multi-git awareness when Project is outer root

When a project folder contains multiple git repositories:

1. Discover nested/sibling git roots (depth-limited scan + manual pin/exclude).
2. Do **not** assume outer git worktree is the only useful isolation unit.
3. File tree of a “feature workspace” should still present the logical project tree users expect, or clearly compose mounts from:
- outer docs/rules checkout or shared view
- product worktree(s)

### P0 — “New workspace” should ask *which git target(s)* to isolate

Composer UX:

- Project: `CliProxy` (filesystem root)
- Isolate for this task:
- [ ] outer meta repo
- [x] `CliRelay`
- [x] `codeProxy`
- Context policy:
- include outer `AGENTS.md` / `docs` / `rules` always (default on)
- include sibling product repos read-only (default on for cross-service)

If only one product repo is selected, still keep outer governance files visible.

### P1 — Workspaces decoupled from automatic outer `git worktree add`

This overlaps #2654, but with nested-repo specifics:

- Allow a workspace that is a **composed view** rather than a pure worktree of project root.
- Allow attaching existing product worktrees created outside Orca.
- Folder/meta projects should not be second-class after the first synthetic workspace.

### P1 — Per-repo SCM + PR, shared agent context

One agent conversation may touch:

- docs in outer repo
- backend worktree
- frontend worktree

UI should show multiple SCM lanes / PR states without forcing separate Orca projects that fragment chat/context.

### P2 — Explicit support for “ignored nested checkouts”

Outer `.gitignore` of product directories is common and correct.

Orca should treat those paths as **workspace members + independent git targets**, not as “non-existent in worktree because not in outer index.”

### P2 — Agent instructions path resolution across roots

If workspace has outer `AGENTS.md` and inner `AGENTS.md`, define precedence and always load outer governance when project root is outer.

---

## Suggested UX flows

### Flow 1 — Single product feature (backend only)

1. Project remains `~/.../CliProxy`
2. User starts “New workspace / agent”
3. Orca creates `CliRelay` worktree on branch `feat/x`
4. Agent context includes:
- outer `AGENTS.md`, `docs/`, `rules/`
- `CliRelay` worktree files
- optional read-only `codeProxy` main checkout for API contract reference
5. Commits/PRs only against `CliRelay`

### Flow 2 — Cross-service feature

1. Same project root
2. Orca creates two product worktrees (`CliRelay`, `codeProxy`) linked as one task group
3. Shared agent context across both + outer docs
4. Separate PRs per repo, one orchestration surface in Orca

### Flow 3 — Docs-only investigation

1. Isolate outer meta repo only (or no worktree, just branch/worktree on outer)
2. Product repos remain read-only mounts
3. Agent writes plan/review under `docs/` without touching product histories

These three flows are all first-class in human workflows; only Flow-with-single-product-root is smooth in Orca today, and even that loses outer context if you switch project root.

---

## Non-goals / clarifications

This request is **not** asking Orca to abandon parallel worktrees. Worktrees are great.

It is asking to stop using **outer-repo worktree creation** as the universal definition of “workspace” for multi-git projects.

Also not asking for:

- forcing users into classical monorepos
- requiring git submodules (submodules are a different, often worse tool for this)
- a pure “open any random folders” mode without git awareness (though multi-root helps)

The ideal is: **worktree power + multi-root intelligence**.

---

## Acceptance criteria

A nested meta+product layout like the one above should satisfy:

1. Opening the outer folder as Project preserves full logical file visibility for agents (docs + product checkouts).
2. Creating a new agent workspace does **not** strand the user with docs-only worktree when product dirs are independent git repos ignored by outer git.
3. User can select one or more product git targets for isolation.
4. Outer governance files remain in agent context even when write isolation is product-scoped.
5. SCM/PR operations are scoped to the correct git target(s).
6. Cross-service tasks can attach multiple product worktrees under one task/project without splitting agent memory.
7. Behavior is documented with an official “meta-repo / multi-repo project” guide.

---

## Why this matters to Orca specifically

Orca markets itself as the ADE for fleets of parallel agents. Parallelism multiplies whatever model you choose:

- If the model is single-root worktree, fleets are excellent inside one repo.
- If real teams run meta-root multi-repo systems, fleets become excellent at generating **wrong-context work** quickly.

Several existing issues (#7900, #1099, #7568, #2654, #8675) already show demand for multi-repo. This issue adds the missing diagnosis for a very common AI-era layout:

> **Context root ≠ git isolation root**

Until Orca first-classes that split, nested multi-repo users will keep bouncing between:

- Orca (great orchestration, wrong workspace model)
- and tools like Zed/VS Code/other agent runners (correct multi-root context, weaker fleet UX)

---

## Environment / evidence notes

- Platform observed: macOS desktop Orca
- Project shape: outer git meta-repo + 2 nested independent product repos ignored by outer `.gitignore`
- Failure: “New workspace” from outer project yields docs/rules-only tree; product sources not present as expected
- Workaround attempted: open product repo as project → fixes isolation, causes agent instruction/doc isolation
- Comparison: same folder opened in multi-git-aware editors keeps both context and per-repo SCM

Happy to provide a minimal public fixture repo pair if useful for QA:

- `demo-meta/` (outer git with AGENTS.md + ignored children)
- `demo-meta/api/` and `demo-meta/web/` as independent repos

---

## Ask to maintainers

Please consider promoting this from scattered multi-repo feature requests into an explicit platform principle:

1. **Workspace filesystem** for agent context
2. **Git targets** for isolation/PR
3. **Composable mounts** for nested/sibling repos and governance docs

Even a staged rollout helps:

- Stage 1: discover nested git roots + don’t drop ignored product checkouts from project views
- Stage 2: create product-scoped worktrees while keeping outer context mounted
- Stage 3: multi-target task groups with multi-PR awareness

I am not asking users with simple single-repo workflows to lose anything. I am asking Orca not to make multi-repo AI teams contort their project architecture to fit a single-root assumption.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.