github / github/copilot-cli

Startup input lag (15–30 s freeze) since 1.0.52: unbounded recursive glob over $COPILOT_CUSTOM_INSTRUCTIONS_DIRS entries

Abierto
#3,543 1 comentario 0 reacciones 0 asignados Ver en GitHub
area:configuration area:context-memory
Lenguaje dominante
Shell
Estrellas
11.2k
Forks
1.9k
Merge medio
14 h 16 min
PR fusionados (30 d)
6

Descripción

### Describe the bug

When `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` is set to a directory that contains a large subtree (e.g. `$HOME` for users who keep monorepos under their home), `copilot` freezes for **15–30 seconds at startup**. During the freeze the TUI is fully unresponsive: keystrokes are buffered but not echoed, the rendered output stops updating, and `Ctrl-C` does not register until the freeze ends.

Root cause: the instruction-discovery code in 1.0.52+ does an unbounded recursive `glob("$dir/**/*.instructions.md", { nocase: true })` for each entry in `COPILOT_CUSTOM_INSTRUCTIONS_DIRS`, with **no depth limit** and **no skip list** for `node_modules`/`.git`/`vendor`/build output dirs. On a realistic `$HOME` this fires 600,000+ `readdir` callbacks back-to-back on the main event loop.

The regression has persisted in every release from `1.0.52` (final) through `1.0.55-5` (current latest pre-release, as of 2026-05-27). It is **not** present in any `1.0.52-N` pre-release (`1.0.52-0` through `1.0.52-4` are all clean), nor in `1.0.51` or `1.0.51-N`.

**Confirmed isolation:** unsetting `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` fully fixes startup on `1.0.55-5` (16 s wall / 30 ms worst keystroke vs 70 s / 22 s with the var set).

### Affected version

GitHub Copilot CLI 1.0.55-5

### Steps to reproduce the behavior

The minimum reliable repro on any broken version:

```bash
# Baseline (instant):
copilot --no-auto-update

# Broken (15–30 s freeze before the prompt is usable):
COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME copilot --no-auto-update
```

Severity scales with `$HOME` size. A `$HOME` with only dotfiles won't freeze noticeably; a `$HOME` with monorepos / `node_modules` / Cargo build artifacts hangs for 15–30 s and keystrokes typed during the window do not echo until it ends. Setting the env var to e.g. `/` would be even worse.

A programmatic harness that drives the same flow non-interactively and measures keystroke latency to ms precision is at the end of this report.

### Expected behavior

The prompt should be responsive to keystrokes immediately after startup (input latency in the tens of milliseconds, not tens of seconds). Setting `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` should not cause a 15–30 s freeze proportional to the size of the directory tree. Even if that directory tree is known to be large it should not freeze the main UI thread.

### Additional context

### Environment

- **OS:** WSL2 Ubuntu 24.04 on Windows 11
- **Arch:** x86_64
- **Terminal:** Windows Terminal
- **Multiplexer:** tmux 3.4 (`TERM_PROGRAM=tmux`)
- **TERM:** `xterm-color`
- **Shell:** zsh
- **Account type:** GitHub Enterprise
(`api.enterprise.githubcopilot.com`)
- **Env var triggering the bug:** `COPILOT_CUSTOM_INSTRUCTIONS_DIRS=/home/rsiemens`
- **$HOME size:** ~hundreds of GB, contains multi-million-file monorepos

### Measured timings on this system

Two back-to-back runs per version, same network, same `$HOME`, same minute. `wall(s)` is from spawn to "responsive again"; `ls_resp(s)` is the wall delay between sending `!ls` and seeing its output; `worst_probe(ms)` is the largest single-keystroke latency observed during the post-`Ctrl-C` probe window.

| version | wall (s) | `!ls` response (s) | worst probe (ms) | status |
| -------- | -------: | ---------------: | ---------------: | -------- |
| 1.0.51-2 | 11.62 | 0.15 | 28 | OK |
| 1.0.51-3 | 11.56 | 0.15 | 34 | OK |
| 1.0.51 | 11.58 | 0.23 | 34 | OK |
| 1.0.52-0 | 11.08 | 0.17 | 44 | OK |
| 1.0.52-1 | 10.94 | 0.18 | 44 | OK |
| 1.0.52-2 | 11.74 | 0.22 | 48 | OK |
| 1.0.52-3 | 11.72 | 0.25 | 32 | OK |
| 1.0.52-4 | 12.00 | 0.22 | 40 | OK |
| **1.0.52** | **77.22** | **14.57** | **15,807** | **BROKEN** |
| 1.0.53-0 | 64.15 | 15.68 | 25,819 | BROKEN |
| 1.0.53-1 | 64.00 | 15.56 | 23,054 | BROKEN |
| 1.0.53-2 | 65.10 | 14.51 | 24,019 | BROKEN |
| 1.0.53 | 79.51 | 15.34 | 36,197 | BROKEN |
| 1.0.54 | 65.23 | 15.63 | 24,024 | BROKEN |
| 1.0.55-0 | 62.80 | 14.48 | 21,386 | BROKEN |
| 1.0.55-1 | 60.52 | 14.91 | 17,829 | BROKEN |
| 1.0.55-2 | 68.41 | 17.89 | 18,937 | BROKEN |
| 1.0.55-3 | 80.68 | 20.55 | 32,825 | BROKEN |
| 1.0.55-4 | 76.28 | 20.58 | 26,195 | BROKEN |
| **1.0.55-5** | **69.99** | **19.40** | **22,557** | **BROKEN (current latest)** |

The variance among the broken builds is run-to-run noise (which sub-tree of `$HOME` happens to be walked first). The qualitative step-function at the 1.0.52-4 → 1.0.52 boundary is solid.

### Isolation experiment

All against `1.0.55-5`, same `$HOME` snapshot, same minute:

| condition | wall | worst keystroke | bug present? |
| ------------------------------------------------------------------------ | ----: | --------------: | -----------: |
| env var set to `/home/rsiemens` | 70 s | 22,557 ms | yes |
| env var set, empty cwd, real `~/.copilot` | 80 s | 27,402 ms | yes |
| env var set, real cwd, empty `~/.copilot` (no sessions) | 80 s | 28,362 ms | yes |
| **env var unset** | 16 s | 30 ms | **no** |

Only the env var matters; nothing else does.

### Root cause

In the minified app bundle (1.0.55-5), the instruction-source loader is:

```js
// ixi(workingDir, gitRoot, additionalDirs)
async function ixi(t, e, r = []) {
let o = SHt([t, ...(e && !m0(e, t) ? [e] : [])]); // [cwd, gitRoot]
let s = SHt(r); // additionalDirs
let a = await Promise.all([
...o.map(l => Oet(Ui.join(l, ".github", "instructions"), l)), // OK — scoped
...s.map(l => Oet(Ui.join(l, ".github", "instructions"), l)), // OK — scoped
...s.map(l => Oet(l, l)), // BUG — unbounded
]);
...
}

async function lxi(t) {
let e = Let(Ui.join(t, "**", "*.instructions.md"));
return OG(e, { nocase: !0 }); // glob(): no maxDepth, no skip list
}
```

`additionalDirs` is `[...COPILOT_CUSTOM_INSTRUCTIONS_DIRS_split_on_comma, ...settings.additionalDirs]`. The third map line walks every additional dir as a recursive glob root, instead of joining it with `.github/instructions/` like the first two map lines do.

Sibling code in the same module *is* bounded: the AGENTS.md / CLAUDE.md / GEMINI.md discovery BFS (`fWr`) uses `maxDepth=2` and skips `["node_modules", ".git", "vendor", "dist", "build", ".next", ".nuxt", "out", "coverage"]`. The `child_git_repo_scan` BFS (`w9n`) uses `maxDepth=5`, a 2 s timeout, and a 1000-directory cap. Only `lxi` is unbounded.

### Suggested fixes

In order of preference:

1. **Drop the third `Oet(l, l)` branch in `ixi`.** Users who set `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` almost certainly expect it to behave like the cwd/git-root branches (look in `.github/instructions/` only). The unbounded `**/*.instructions.md` fallback is a footgun with no documented purpose.
2. **Bound `lxi`** with the same skip-set already used by `fWr` (the `jIi = ["node_modules", ".git", ...]` set) and a reasonable depth limit (depth 3 would have prevented this).
3. **Validate `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` entries against a sanity check** (reject roots that have > N children or contain known massive-tree markers like `node_modules`).
4. **Move instruction discovery off the startup critical path** entirely — discover lazily on first use, or yield to the event loop between readdirs.

### Reproduction harness

`measure.py` — drives a fresh `copilot` in a pty, sends `!ls` + Enter after settle, sends `Ctrl-C` as soon as ls output appears, then probes with single-byte keystrokes at 250 ms cadence and records the latency to next byte of output. Run it once with `COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME` and once with it unset, on the same binary, for a one-shot demonstration.

```python
#!/usr/bin/env python3
"""Reproduce the Copilot CLI input-latency regression.
Run:
COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME python3 measure.py /path/to/copilot
unset COPILOT_CUSTOM_INSTRUCTIONS_DIRS; python3 measure.py /path/to/copilot
"""
import fcntl, json, os, pty, re, select, signal, struct, sys, termios, time

ANSI = re.compile(rb'\x1b\[[0-9;?]*[a-zA-Z]|\x1b\][^\x07]*\x07|\x1b[@-Z\\-_]')

def spawn(argv, env, cwd, rows, cols):
pid, fd = pty.fork()
if pid == 0:
os.chdir(cwd)
fcntl.ioctl(0, termios.TIOCSWINSZ, struct.pack("HHHH", rows, cols, 0, 0))
os.execvpe(argv[0], argv, env)
os._exit(127)
fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", rows, cols, 0, 0))
return pid, fd

def main(binary):
env = dict(os.environ)
for k in ("COPILOT_AGENT_SESSION_ID","COPILOT_CLI","COPILOT_LOADER_PID",
"COPILOT_RUN_APP","COPILOT_CLI_BINARY_VERSION"):
env.pop(k, None)
env.setdefault("TERM", "xterm-color")
env.setdefault("COLUMNS", "237"); env.setdefault("LINES", "63")
t0 = time.monotonic()
pid, fd = spawn([binary, "--no-auto-update"], env, os.getcwd(), 63, 237)
poller = select.poll(); poller.register(fd, select.POLLIN|select.POLLHUP|select.POLLERR)
blob = bytearray(); events = []; probes = []
state = "settle"; last_out = t0; last_probe = None; next_probe = None
ls_sent = ls_seen = ctrlc_sent = None
end = t0 + 120
while time.monotonic() < end:
t = time.monotonic(); el = t - t0
if state == "settle" and (t - last_out) >= 4.0 and el > 5:
os.write(fd, b"!ls\r"); ls_sent = el
state = "await_ls"; ls_deadline = t + 30
events.append((el, "ls_sent"))
elif state == "await_ls":
stripped = ANSI.sub(b"", bytes(blob))[-4000:]
if any(m in stripped for m in (b"\nscripts", b"scripts\n",
b"README", b"\n.git", b"\n.")):
os.write(fd, b"\x03"); ctrlc_sent = el; ls_seen = el
state = "probe"; next_probe = t + 0.25
events.append((el, "ls_seen_and_ctrlc"))
elif t > ls_deadline:
os.write(fd, b"\x03"); ctrlc_sent = el
state = "probe"; next_probe = t + 0.25
events.append((el, "ls_timeout_and_ctrlc"))
elif state == "probe":
if next_probe and t >= next_probe and last_probe is None:
os.write(fd, b" "); last_probe = t
if len(probes) >= 12: break
for fd_, ev in poller.poll(20):
if ev & (select.POLLIN|select.POLLPRI):
d = os.read(fd, 65536)
if not d: end = t; break
blob += d; last_out = time.monotonic()
if last_probe is not None:
gap = (last_out - last_probe) * 1000
if gap >= 5:
probes.append(gap); last_probe = None
next_probe = last_out + 0.25
if ev & (select.POLLHUP|select.POLLERR):
end = t
try: os.kill(pid, signal.SIGTERM)
except ProcessLookupError: pass
print(json.dumps({
"binary": binary,
"env_var": env.get("COPILOT_CUSTOM_INSTRUCTIONS_DIRS"),
"wall_sec": round(time.monotonic()-t0, 3),
"ls_response_sec": (None if not (ls_sent and ls_seen) else round(ls_seen-ls_sent, 3)),
"probe_latencies_ms": [round(p,1) for p in probes],
"max_probe_ms": (round(max(probes),1) if probes else None),
}, indent=2))

if __name__ == "__main__":
main(sys.argv[1])
```

Expected output (on a large `$HOME`, broken version):

```
$ unset COPILOT_CUSTOM_INSTRUCTIONS_DIRS
$ python3 measure.py ./copilot-1.0.55-5
{ "wall_sec": 15.9, "ls_response_sec": 0.21, "max_probe_ms": 30.0, ... }

$ COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME python3 measure.py ./copilot-1.0.55-5
{ "wall_sec": 70.0, "ls_response_sec": 19.40, "max_probe_ms": 22557.0, ... }
```

### Workaround for affected users

Either of:

```bash
unset COPILOT_CUSTOM_INSTRUCTIONS_DIRS
# or
export COPILOT_CUSTOM_INSTRUCTIONS_DIRS=/some/small/dir/with/instructions
```

Pinning to `1.0.52-4` also works:

```bash
npm install -g @github/copilot@1.0.52-4
```

### Possibly-related existing issues

- [#3408](https://github.com/github/copilot-cli/issues/3408) (WSL startup lag in 1.0.49 — different bug; that one was in a 1.0.49 prerelease block and was fixed before 1.0.49 final).
- [#3439](https://github.com/github/copilot-cli/issues/3439) (tmux on mintty/Cygwin rendering lag — different environment but possibly related symptom).

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.