github / github/copilot-cli

CLI headless mode: internal state accumulation causes progressive latency degradation across sessions

Open
#2,755 0 comments 0 reactions 0 assignees View on GitHub
area:non-interactive area:sessions
Dominant language
Shell
Stars
11.2k
Forks
1.9k
Avg merge
14h 16m
Merged PRs (30d)
6

Description

### Describe the bug

# CLI headless mode: internal state accumulation causes progressive latency degradation across sessions

## Summary

When running the Copilot CLI in headless mode (`copilot --headless --port `) with BYOK (Azure OpenAI), **response latency degrades progressively with each new session**, even after properly disconnecting and deleting sessions via the SDK. The first request after a fresh CLI start completes in ~1–3s, but subsequent requests degrade to 17–30s. Only killing and restarting the CLI process restores performance.

## Environment

- **CLI version**: 1.0.28
- **SDK version (Python)**: 0.2.2 (container), 0.1.32 (host)
- **OS**: Debian (python:3.13-slim Docker image), Linux amd64
- **Provider**: Azure OpenAI (BYOK) — `gpt-4o-mini` deployment
- **Provider wire API**: `responses`
- **Azure OpenAI API version**: `2025-01-01-preview`

## Steps to Reproduce

1. Start CLI in headless mode inside a container:
```bash
copilot --headless --port 14321
```

2. From an external process, connect via SDK, create a session, send a message, disconnect, delete the session:
```python
client = CopilotClient({"cli_url": "container-host:4321"})
await client.start()

session = await client.create_session(
session_id="test-1",
model="gpt-4o-mini",
provider={"type": "azure", "base_url": "...", "api_key": "...", "azure": {"api_version": "..."}},
system_message={"mode": "replace", "content": "Answer briefly."},
streaming=True,
)
response = await session.send_and_wait("What is 2+2?") # ~1-3s ✅
await session.disconnect()

# Cleanup
sessions = await client.list_sessions()
for s in sessions:
await client.delete_session(s.session_id)
await client.stop()
```

3. From a **new process** (fresh Python interpreter, new `CopilotClient` instance), repeat step 2 with a different `session_id`:
```python
# New process, new client, new session_id
response = await session.send_and_wait("What is 2+2?") # ~17-30s ❌
```

4. Each subsequent new-process request gets progressively slower, stabilizing at ~28-30s.

## Expected Behavior

After `session.disconnect()` + `client.delete_session()`, the CLI should fully release all resources associated with that session. New sessions — whether from the same or a different SDK client process — should have consistent latency (~1-3s for a simple prompt when the LLM backend responds in <1s).

## Actual Behavior

| Request # | Same CLI instance | `send_and_wait` latency | Notes |
|-----------|-------------------|------------------------|-------|
| 1 | Fresh start | ~1,000–3,000 ms | ✅ Fast |
| 2 | Reused | ~17,000 ms | ❌ Degraded |
| 3 | Reused | ~28,000–30,000 ms | ❌ Severely degraded |
| 4+ | Reused | ~28,000–30,000 ms | ❌ Plateaus at ~29s |
| 1 (after kill+restart) | Fresh start | ~1,000–3,000 ms | ✅ Fast again |

**Key observations:**

- **Azure OpenAI is NOT the bottleneck**: Direct HTTP calls to the same Azure endpoint consistently return in ~900ms (verified with httpx, bypassing CLI/SDK entirely).
- **`delete_session()` does not fully clean up**: After `list_sessions()` + `delete_session()` for all sessions, `list_sessions()` returns empty, BUT the CLI process still retains internal state that causes degradation.
- **Wiping session-state files doesn't help**: Deleting `/root/.copilot/session-state/*` on disk while the CLI is running has no effect — the state is held in memory.
- **Only a process kill fixes it**: `killall copilot` followed by a fresh `copilot --headless --port ` restores full performance immediately.
- **Pattern is consistent**: Reproduced dozens of times across multiple hours. The degradation occurs even when each request uses a unique `session_id` and a completely fresh `CopilotClient` from a new OS process.

## Workaround (current)

We run the CLI via an entrypoint script that auto-restarts it when killed. After each SDK request, the application kills the CLI process via `docker exec killall copilot`, and the entrypoint restarts it with clean state. This is fragile and adds ~3-5s of restart overhead per request.

```bash
# entrypoint.sh (simplified)
copilot --headless --port 14321 &
CLI_PID=$!
while true; do
if ! kill -0 $CLI_PID 2>/dev/null; then
rm -rf /root/.copilot/session-state/*
copilot --headless --port 14321 &
CLI_PID=$!
fi
sleep 1
done
```

## Impact

This makes the CLI unsuitable for any multi-request server workload (web backends, API services, chatbots) without the kill-restart hack. In production, our chat feature degrades from sub-3s responses to 30s responses after just 2 user messages.

## Possible Root Cause (speculation)

The CLI appears to retain conversation context or model state in memory across sessions even after `delete_session()`. This accumulated context may be sent with each new request to the LLM provider, causing the provider to process increasingly large payloads (explaining the progressive slowdown). The ~29s plateau could be the provider's timeout or max-context processing time.

## Reproduction Script

Full reproduction script available: creates N sequential requests with fresh SDK clients, measures latency, and optionally kills CLI between requests to demonstrate the fix.

```python
# Set environment variables:
# COPILOT_CLI_URL=localhost:4321
# AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
# AZURE_OPENAI_API_KEY=your-key
# AZURE_OPENAI_API_VERSION=2025-01-01-preview
# NUM_REQUESTS=3
# KILL_CLI=0 (set to 1 to kill CLI between requests — makes all fast)

python tools/test_sdk_perf.py
```

[test_sdk_perf.py](https://github.com/user-attachments/files/26781601/test_sdk_perf.py)

### Affected version

_No response_

### Steps to reproduce the behavior

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.