Startup input lag (15–30 s freeze) since 1.0.52: unbounded recursive glob over $COPILOT_CUSTOM_INSTRUCTIONS_DIRS entries
- Vorherrschende Sprache
- Shell
- Sterne
- 11.2k
- Forks
- 1.9k
- Ø Merge
- 14 Std. 16 Min.
- Gemergte PRs (30 T.)
- 6
Beschreibung
### Describe the bug
When `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` is set to a directory that contains a large subtree (e.g. `$HOME` for users who keep monorepos under their home), `copilot` freezes for **15–30 seconds at startup**. During the freeze the TUI is fully unresponsive: keystrokes are buffered but not echoed, the rendered output stops updating, and `Ctrl-C` does not register until the freeze ends.
Root cause: the instruction-discovery code in 1.0.52+ does an unbounded recursive `glob("$dir/**/*.instructions.md", { nocase: true })` for each entry in `COPILOT_CUSTOM_INSTRUCTIONS_DIRS`, with **no depth limit** and **no skip list** for `node_modules`/`.git`/`vendor`/build output dirs. On a realistic `$HOME` this fires 600,000+ `readdir` callbacks back-to-back on the main event loop.
The regression has persisted in every release from `1.0.52` (final) through `1.0.55-5` (current latest pre-release, as of 2026-05-27). It is **not** present in any `1.0.52-N` pre-release (`1.0.52-0` through `1.0.52-4` are all clean), nor in `1.0.51` or `1.0.51-N`.
**Confirmed isolation:** unsetting `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` fully fixes startup on `1.0.55-5` (16 s wall / 30 ms worst keystroke vs 70 s / 22 s with the var set).
### Affected version
GitHub Copilot CLI 1.0.55-5
### Steps to reproduce the behavior
The minimum reliable repro on any broken version:
```bash
# Baseline (instant):
copilot --no-auto-update
# Broken (15–30 s freeze before the prompt is usable):
COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME copilot --no-auto-update
```
Severity scales with `$HOME` size. A `$HOME` with only dotfiles won't freeze noticeably; a `$HOME` with monorepos / `node_modules` / Cargo build artifacts hangs for 15–30 s and keystrokes typed during the window do not echo until it ends. Setting the env var to e.g. `/` would be even worse.
A programmatic harness that drives the same flow non-interactively and measures keystroke latency to ms precision is at the end of this report.
### Expected behavior
The prompt should be responsive to keystrokes immediately after startup (input latency in the tens of milliseconds, not tens of seconds). Setting `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` should not cause a 15–30 s freeze proportional to the size of the directory tree. Even if that directory tree is known to be large it should not freeze the main UI thread.
### Additional context
### Environment
- **OS:** WSL2 Ubuntu 24.04 on Windows 11
- **Arch:** x86_64
- **Terminal:** Windows Terminal
- **Multiplexer:** tmux 3.4 (`TERM_PROGRAM=tmux`)
- **TERM:** `xterm-color`
- **Shell:** zsh
- **Account type:** GitHub Enterprise
(`api.enterprise.githubcopilot.com`)
- **Env var triggering the bug:** `COPILOT_CUSTOM_INSTRUCTIONS_DIRS=/home/rsiemens`
- **$HOME size:** ~hundreds of GB, contains multi-million-file monorepos
### Measured timings on this system
Two back-to-back runs per version, same network, same `$HOME`, same minute. `wall(s)` is from spawn to "responsive again"; `ls_resp(s)` is the wall delay between sending `!ls` and seeing its output; `worst_probe(ms)` is the largest single-keystroke latency observed during the post-`Ctrl-C` probe window.
| version | wall (s) | `!ls` response (s) | worst probe (ms) | status |
| -------- | -------: | ---------------: | ---------------: | -------- |
| 1.0.51-2 | 11.62 | 0.15 | 28 | OK |
| 1.0.51-3 | 11.56 | 0.15 | 34 | OK |
| 1.0.51 | 11.58 | 0.23 | 34 | OK |
| 1.0.52-0 | 11.08 | 0.17 | 44 | OK |
| 1.0.52-1 | 10.94 | 0.18 | 44 | OK |
| 1.0.52-2 | 11.74 | 0.22 | 48 | OK |
| 1.0.52-3 | 11.72 | 0.25 | 32 | OK |
| 1.0.52-4 | 12.00 | 0.22 | 40 | OK |
| **1.0.52** | **77.22** | **14.57** | **15,807** | **BROKEN** |
| 1.0.53-0 | 64.15 | 15.68 | 25,819 | BROKEN |
| 1.0.53-1 | 64.00 | 15.56 | 23,054 | BROKEN |
| 1.0.53-2 | 65.10 | 14.51 | 24,019 | BROKEN |
| 1.0.53 | 79.51 | 15.34 | 36,197 | BROKEN |
| 1.0.54 | 65.23 | 15.63 | 24,024 | BROKEN |
| 1.0.55-0 | 62.80 | 14.48 | 21,386 | BROKEN |
| 1.0.55-1 | 60.52 | 14.91 | 17,829 | BROKEN |
| 1.0.55-2 | 68.41 | 17.89 | 18,937 | BROKEN |
| 1.0.55-3 | 80.68 | 20.55 | 32,825 | BROKEN |
| 1.0.55-4 | 76.28 | 20.58 | 26,195 | BROKEN |
| **1.0.55-5** | **69.99** | **19.40** | **22,557** | **BROKEN (current latest)** |
The variance among the broken builds is run-to-run noise (which sub-tree of `$HOME` happens to be walked first). The qualitative step-function at the 1.0.52-4 → 1.0.52 boundary is solid.
### Isolation experiment
All against `1.0.55-5`, same `$HOME` snapshot, same minute:
| condition | wall | worst keystroke | bug present? |
| ------------------------------------------------------------------------ | ----: | --------------: | -----------: |
| env var set to `/home/rsiemens` | 70 s | 22,557 ms | yes |
| env var set, empty cwd, real `~/.copilot` | 80 s | 27,402 ms | yes |
| env var set, real cwd, empty `~/.copilot` (no sessions) | 80 s | 28,362 ms | yes |
| **env var unset** | 16 s | 30 ms | **no** |
Only the env var matters; nothing else does.
### Root cause
In the minified app bundle (1.0.55-5), the instruction-source loader is:
```js
// ixi(workingDir, gitRoot, additionalDirs)
async function ixi(t, e, r = []) {
let o = SHt([t, ...(e && !m0(e, t) ? [e] : [])]); // [cwd, gitRoot]
let s = SHt(r); // additionalDirs
let a = await Promise.all([
...o.map(l => Oet(Ui.join(l, ".github", "instructions"), l)), // OK — scoped
...s.map(l => Oet(Ui.join(l, ".github", "instructions"), l)), // OK — scoped
...s.map(l => Oet(l, l)), // BUG — unbounded
]);
...
}
async function lxi(t) {
let e = Let(Ui.join(t, "**", "*.instructions.md"));
return OG(e, { nocase: !0 }); // glob(): no maxDepth, no skip list
}
```
`additionalDirs` is `[...COPILOT_CUSTOM_INSTRUCTIONS_DIRS_split_on_comma, ...settings.additionalDirs]`. The third map line walks every additional dir as a recursive glob root, instead of joining it with `.github/instructions/` like the first two map lines do.
Sibling code in the same module *is* bounded: the AGENTS.md / CLAUDE.md / GEMINI.md discovery BFS (`fWr`) uses `maxDepth=2` and skips `["node_modules", ".git", "vendor", "dist", "build", ".next", ".nuxt", "out", "coverage"]`. The `child_git_repo_scan` BFS (`w9n`) uses `maxDepth=5`, a 2 s timeout, and a 1000-directory cap. Only `lxi` is unbounded.
### Suggested fixes
In order of preference:
1. **Drop the third `Oet(l, l)` branch in `ixi`.** Users who set `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` almost certainly expect it to behave like the cwd/git-root branches (look in `.github/instructions/` only). The unbounded `**/*.instructions.md` fallback is a footgun with no documented purpose.
2. **Bound `lxi`** with the same skip-set already used by `fWr` (the `jIi = ["node_modules", ".git", ...]` set) and a reasonable depth limit (depth 3 would have prevented this).
3. **Validate `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` entries against a sanity check** (reject roots that have > N children or contain known massive-tree markers like `node_modules`).
4. **Move instruction discovery off the startup critical path** entirely — discover lazily on first use, or yield to the event loop between readdirs.
### Reproduction harness
`measure.py` — drives a fresh `copilot` in a pty, sends `!ls` + Enter after settle, sends `Ctrl-C` as soon as ls output appears, then probes with single-byte keystrokes at 250 ms cadence and records the latency to next byte of output. Run it once with `COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME` and once with it unset, on the same binary, for a one-shot demonstration.
```python
#!/usr/bin/env python3
"""Reproduce the Copilot CLI input-latency regression.
Run:
COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME python3 measure.py /path/to/copilot
unset COPILOT_CUSTOM_INSTRUCTIONS_DIRS; python3 measure.py /path/to/copilot
"""
import fcntl, json, os, pty, re, select, signal, struct, sys, termios, time
ANSI = re.compile(rb'\x1b\[[0-9;?]*[a-zA-Z]|\x1b\][^\x07]*\x07|\x1b[@-Z\\-_]')
def spawn(argv, env, cwd, rows, cols):
pid, fd = pty.fork()
if pid == 0:
os.chdir(cwd)
fcntl.ioctl(0, termios.TIOCSWINSZ, struct.pack("HHHH", rows, cols, 0, 0))
os.execvpe(argv[0], argv, env)
os._exit(127)
fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", rows, cols, 0, 0))
return pid, fd
def main(binary):
env = dict(os.environ)
for k in ("COPILOT_AGENT_SESSION_ID","COPILOT_CLI","COPILOT_LOADER_PID",
"COPILOT_RUN_APP","COPILOT_CLI_BINARY_VERSION"):
env.pop(k, None)
env.setdefault("TERM", "xterm-color")
env.setdefault("COLUMNS", "237"); env.setdefault("LINES", "63")
t0 = time.monotonic()
pid, fd = spawn([binary, "--no-auto-update"], env, os.getcwd(), 63, 237)
poller = select.poll(); poller.register(fd, select.POLLIN|select.POLLHUP|select.POLLERR)
blob = bytearray(); events = []; probes = []
state = "settle"; last_out = t0; last_probe = None; next_probe = None
ls_sent = ls_seen = ctrlc_sent = None
end = t0 + 120
while time.monotonic() < end:
t = time.monotonic(); el = t - t0
if state == "settle" and (t - last_out) >= 4.0 and el > 5:
os.write(fd, b"!ls\r"); ls_sent = el
state = "await_ls"; ls_deadline = t + 30
events.append((el, "ls_sent"))
elif state == "await_ls":
stripped = ANSI.sub(b"", bytes(blob))[-4000:]
if any(m in stripped for m in (b"\nscripts", b"scripts\n",
b"README", b"\n.git", b"\n.")):
os.write(fd, b"\x03"); ctrlc_sent = el; ls_seen = el
state = "probe"; next_probe = t + 0.25
events.append((el, "ls_seen_and_ctrlc"))
elif t > ls_deadline:
os.write(fd, b"\x03"); ctrlc_sent = el
state = "probe"; next_probe = t + 0.25
events.append((el, "ls_timeout_and_ctrlc"))
elif state == "probe":
if next_probe and t >= next_probe and last_probe is None:
os.write(fd, b" "); last_probe = t
if len(probes) >= 12: break
for fd_, ev in poller.poll(20):
if ev & (select.POLLIN|select.POLLPRI):
d = os.read(fd, 65536)
if not d: end = t; break
blob += d; last_out = time.monotonic()
if last_probe is not None:
gap = (last_out - last_probe) * 1000
if gap >= 5:
probes.append(gap); last_probe = None
next_probe = last_out + 0.25
if ev & (select.POLLHUP|select.POLLERR):
end = t
try: os.kill(pid, signal.SIGTERM)
except ProcessLookupError: pass
print(json.dumps({
"binary": binary,
"env_var": env.get("COPILOT_CUSTOM_INSTRUCTIONS_DIRS"),
"wall_sec": round(time.monotonic()-t0, 3),
"ls_response_sec": (None if not (ls_sent and ls_seen) else round(ls_seen-ls_sent, 3)),
"probe_latencies_ms": [round(p,1) for p in probes],
"max_probe_ms": (round(max(probes),1) if probes else None),
}, indent=2))
if __name__ == "__main__":
main(sys.argv[1])
```
Expected output (on a large `$HOME`, broken version):
```
$ unset COPILOT_CUSTOM_INSTRUCTIONS_DIRS
$ python3 measure.py ./copilot-1.0.55-5
{ "wall_sec": 15.9, "ls_response_sec": 0.21, "max_probe_ms": 30.0, ... }
$ COPILOT_CUSTOM_INSTRUCTIONS_DIRS=$HOME python3 measure.py ./copilot-1.0.55-5
{ "wall_sec": 70.0, "ls_response_sec": 19.40, "max_probe_ms": 22557.0, ... }
```
### Workaround for affected users
Either of:
```bash
unset COPILOT_CUSTOM_INSTRUCTIONS_DIRS
# or
export COPILOT_CUSTOM_INSTRUCTIONS_DIRS=/some/small/dir/with/instructions
```
Pinning to `1.0.52-4` also works:
```bash
npm install -g @github/copilot@1.0.52-4
```
### Possibly-related existing issues
- [#3408](https://github.com/github/copilot-cli/issues/3408) (WSL startup lag in 1.0.49 — different bug; that one was in a 1.0.49 prerelease block and was fixed before 1.0.49 final).
- [#3439](https://github.com/github/copilot-cli/issues/3439) (tmux on mintty/Cygwin rendering lag — different environment but possibly related symptom).
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.