Investigate intermittent staging Copilot messages first-event timeouts

Background

The staging Copilot POST /copilot/messages endpoint occasionally approaches or exceeds the first-event timeout. The Sentry chart for Copilot messages first event wait P95 shows intermittent staging spikes, including one close to 20 seconds.

An important comparison is that production remains normal during the same staging slow periods, although both environments use the same QNAIGC service endpoint. This makes a global QNAIGC incident alone insufficient to explain the symptom and requires environment-aware evidence.

Evidence screenshot

https://goplus-xk.sentry.io/dashboard/8546578/?project=4509761000308736&project=4509868947800064&statsPeriod=7d

Scope

Locate the source of intermittent first-event latency in staging and determine whether it is caused by:

• an intermittent QNAIGC provider-side failure or slowdown;
• an issue in the network environment hosting builder-backend; or
• a delay inside the backend or its proxy/streaming path after the upstream response begins.

Investigation plan

• Correlate slow staging requests with equivalent production requests over the same time windows, model/provider configuration, and QNAIGC endpoint.
• Add structured telemetry around the outbound QNAIGC request and SSE relay, sufficient to separate DNS/connect/TLS/request-write/response-header/first-upstream-event/first-client-flush timing and classify errors or cancellations.
• Record non-sensitive environment and request correlation fields needed to compare staging and production without logging prompts, response content, credentials, or personal data.
• Correlate the new signals with ingress/proxy and host/network telemetry where available.
• Use traces from slow and normal requests to decide whether the dominant wait is provider-side, network-path-specific, or backend/proxy-side.

Expected outcome

• We can attribute each slow segment with enough confidence to choose the next owner and remediation.
• The added reporting makes future staging-only spikes diagnosable against the production control group.
• Any confirmed configuration, network, provider, or code issue has a separately actionable follow-up or fix.

Related

• Closed predecessor: #3334