github / github/app

Subject: Possible GitHub Copilot SDK artifact-capture/session reliability iss...

Open
#3,515 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
2.1k
Forks
157
PR merge metrics
No merged PRs in 30d

Description

Subject: Possible GitHub Copilot SDK artifact-capture/session reliability issue during long-running certification work
Hi team,
I’d like to bring a possible GitHub Copilot SDK or agent-harness reliability problem onto your radar. We encountered the same failure twice while running a tightly controlled, long-running model certification workflow.
What we are doing
We use GitHub Copilot as the execution host for a code-review certification process. Each certification job:
Creates a fresh Copilot session.
Requests an exact model and reasoning-effort combination.
Gives the model a blind code-review assignment.
Requires the model to submit a structured result through a registered tool named pwi_submit_artifact.
Captures the tool payload, validates it, and persists a receipt containing the observed model identity and artifact hash.
The launcher uses one session at a time, unique artifact paths, explicit timeouts, and immutable attempt directories.
What happened
Two independent attempts stopped with the same failure:
copilot_worker_missing_artifact
In both cases:
A real Copilot session was created.
The requested model and reasoning effort were observed.
The session reached a terminal response.
No pwi_submit_artifact payload was captured by the application.
The same launcher successfully captured artifacts from the preceding sessions.
There were no overlapping session IDs or concurrent workers.
The attempts used different session IDs and different model routes.
Affected sessions:
Attempt 001
Session: 93a97bac-c176-4dab-8e37-031d993c74f1
Observed route: gpt-5.6-luna / low

Attempt 002
Session: 5ad8cf1f-8e07-4e58-9e67-e7ffc55281c3
Observed route: gpt-5.6-sol / low
Across the two attempts, 18 fresh sessions were consumed. The failure occurred on the ninth session of each attempt.
The second occurrence makes us concerned that this is not simply a one-off model response.
What we cannot currently determine
The SDK-facing application tells us that no structured artifact was captured, but the available application evidence does not clearly distinguish among these possibilities:
The model never invoked pwi_submit_artifact.
The model invoked the tool, but the tool event was not delivered or captured.
The model returned the payload in its final response rather than invoking the tool.
The tool invocation occurred, but parsing or SDK event handling failed.
The session terminated before the final tool event was persisted.
Our wrapper currently receives the returned response content, but when structured ArtifactCapture is empty, the operation raises a technical failure. The response/event material available to the wrapper is insufficiently durable to diagnose the boundary conclusively after the fact.
We are examining the local Copilot session-state and process logs for both session IDs. We will provide a redacted diagnostic inventory if it contains useful evidence.
Expected behavior
We would like the SDK or harness to provide durable and unambiguous terminal evidence that distinguishes:
model_did_not_submit_artifact
tool_invocation_observed_but_capture_failed
tool_payload_parse_failed
session_terminated_without_final_response
artifact_validation_failed


Field Value
App version 1.1.14
OS Windows 10.0.26200
Theme Espresso
Path /chat
Tenure Week 3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the local Copilot session-state and process logs for sessions 93a97bac-c176-4dab-8e07-4e58-9e67-e7ffc55281c3 and 5ad8cf1f-8e07-4e67-e7ffc55281c3, then trace how the wrapper handles an empty ArtifactCapture. Done means the SDK or harness records durable terminal evidence distinguishing missing submission, capture or parsing failure, premature termination, and validation failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
github
Domain
desktop, devtools
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.