Eval CI: resolve the copilot-sdk auth blocker -- find an alternative to the PAT (follow-up to #16331)
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
## Background
PR #16331 attempted to move the copilot-sdk Vally auth off the weekly-expiring `azuresdk-copilot-github-pat` PAT and onto a `GH_TOKEN` minted by the shared `login-to-github` step (Azure SDK Automation GitHub App, App ID 1086291). It was closed without merging: `login-to-github` produces a `ghs_` App-installation token, and the copilot-sdk auth docs only list `gho_` / `ghu_` / `github_pat_` as supported token types. We could not confirm before merge that a real `ghs_` token carries Copilot entitlement.
## Goal
Land a durable, non-PAT auth path for the copilot-sdk executor used by the Vally eval pipelines.
## Next steps
- [ ] Run a manual pipeline execution with a real `ghs_` token from `login-to-github` and confirm whether copilot-sdk can authenticate with it / whether it has Copilot entitlement.
- [ ] If entitled: authorize the eval pipeline(s) for the `AzureSDKEngKeyVault Secrets` service connection, unbind `AzSDK_Eval_Variable_group`, and re-land the #16331 changes.
- [ ] If not entitled: evaluate fallback options -- a bot user with an assigned Copilot seat, or copilot-sdk BYOK via Azure OpenAI.
- [ ] Once a durable path is validated, retire the `azuresdk-copilot-github-pat` secret and its variable group.
Related: #16331 (closed, not merged). Part of #16344.
Contributor guide
Research direction
Start with the manual eval pipeline execution and the shared login-to-github step described here, using #16331 for context. Confirm whether its real ghs_ token authenticates copilot-sdk and has Copilot entitlement, then evaluate the listed fallback paths if it does not. Done means a validated durable auth path, updated pipeline authorization, and retirement of the PAT secret and variable group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, github, github-actions
- Domain
- authentication, ci-cd, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100