Azure / Azure/azure-sdk-tools

Eval CI: resolve the copilot-sdk auth blocker -- find an alternative to the PAT (follow-up to #16331)

Open
#16,413 0 comments 0 reactions 0 assignees View on GitHub
needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

## Background

PR #16331 attempted to move the copilot-sdk Vally auth off the weekly-expiring `azuresdk-copilot-github-pat` PAT and onto a `GH_TOKEN` minted by the shared `login-to-github` step (Azure SDK Automation GitHub App, App ID 1086291). It was closed without merging: `login-to-github` produces a `ghs_` App-installation token, and the copilot-sdk auth docs only list `gho_` / `ghu_` / `github_pat_` as supported token types. We could not confirm before merge that a real `ghs_` token carries Copilot entitlement.

## Goal

Land a durable, non-PAT auth path for the copilot-sdk executor used by the Vally eval pipelines.

## Next steps

- [ ] Run a manual pipeline execution with a real `ghs_` token from `login-to-github` and confirm whether copilot-sdk can authenticate with it / whether it has Copilot entitlement.
- [ ] If entitled: authorize the eval pipeline(s) for the `AzureSDKEngKeyVault Secrets` service connection, unbind `AzSDK_Eval_Variable_group`, and re-land the #16331 changes.
- [ ] If not entitled: evaluate fallback options -- a bot user with an assigned Copilot seat, or copilot-sdk BYOK via Azure OpenAI.
- [ ] Once a durable path is validated, retire the `azuresdk-copilot-github-pat` secret and its variable group.

Related: #16331 (closed, not merged). Part of #16344.

Contributor guide

Open the contributing guide

Research direction

Start with the manual eval pipeline execution and the shared login-to-github step described here, using #16331 for context. Confirm whether its real ghs_ token authenticates copilot-sdk and has Copilot entitlement, then evaluate the listed fallback paths if it does not. Done means a validated durable auth path, updated pipeline authorization, and retirement of the PAT secret and variable group.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, github, github-actions
Domain
authentication, ci-cd, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.