[Incident] Deployment failure: aoai-model environment — Go toolchain version mismatch in Smoke Copilot AOAI (Entra) workflow

#### Environment

- **Environment**: `aoai-model`
- **Deployment state**: `failure`
- **Repository**: `github/gh-aw`
- **Failing run**: [Smoke Copilot - AOAI (Entra) #34545423677](https://github.com/github/gh-aw/actions/runs/34545423677) (job: `agent`, step: *Build and install gh-aw CLI from source*)
- **Commit**: `6f90863` — "Apply logs timeout and count globally across concurrent targets (#60058)"
- **Time**: 2026-09-11T00:11:30Z → 00:16:05Z

#### Summary

The `Smoke Copilot - AOAI (Entra)` workflow, which deploys to the `aoai-model` environment, failed during CLI setup with a **Go toolchain version mismatch**. The workflow contains two separate `actions/setup-go` steps within the same job:

1. `Setup Go for CLI build` — uses `go-version-file: go.mod` with `cache: true` → resolves to the exact pinned version **go1.26.7** and populates the Go build cache with objects compiled by that toolchain.
2. `Setup Go` (later in the job, for AWF chroot GOROOT capture) — uses `go-version: '1.26'` with `cache: false` → resolves to the *latest* available 1.26.x, which was **go1.26.8** on the runner image.

When the subsequent "Build and install gh-aw CLI from source" step ran `make build` (`gh extension install .` → `go build ./cmd/gh-aw`), it picked up the go1.26.8 toolchain installed by the second `setup-go` step, but the Go build cache still contained standard-library packages precompiled for go1.26.7. This produced repeated `compile: version "go1.26.8" does not match go tool version "go1.26.7"` errors across dozens of stdlib/vendor packages, and `make build` exited with `Error 1` (step exit code 2).

The failure is unrelated to Azure OpenAI connectivity/credentials — it never reached the AOAI curl call. It is a CI infra/tooling issue in the workflow's Go setup, not a genuine model-endpoint outage.

#### Evidence

Build failure log excerpt (job: agent, step: Build and install gh-aw CLI from source)

```
2026-09-11T00:15:50.6337539Z go build -ldflags "-s -w -X main.version=6f90863-dirty" -o gh-aw ./cmd/gh-aw
2026-09-11T00:15:54.6076183Z # internal/byteorder
2026-09-11T00:15:54.6076183Z compile: version "go1.26.8" does not match go tool version "go1.26.7"
2026-09-11T00:15:54.6089672Z # internal/goarch
2026-09-11T00:15:54.6090483Z compile: version "go1.26.8" does not match go tool version "go1.26.7"
... (repeats for ~35 stdlib/vendor packages, including github.com/github/gh-aw/pkg/setutil) ...
2026-09-11T00:15:54.8803539Z make: *** [Makefile:42: build] Error 1
2026-09-11T00:15:54.8822080Z ##[error]Process completed with exit code 2.
```

Relevant workflow snippet (.github/workflows/smoke-copilot-aoai-entra.lock.yml)

```yaml
- name: Setup Go for CLI build
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: go.mod # resolves to go1.26.7 (go.mod: go 1.26.7)
cache: true
- name: Build gh-aw CLI
run: ... go build ... ./cmd/gh-aw # succeeds, cache populated with go1.26.7 objects
...
- name: Setup Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: '1.26' # resolves to latest 1.26.x -> go1.26.8 on runner
cache: false
...
# Later: "Build and install gh-aw CLI from source" runs `make build` -> fails
```

- No related CI failures precede this — commits immediately before `6f90863` (`c8d513f`, `1a17da9`, `bdc29d9`, `106c257`) landed cleanly. This is an environment/toolchain drift issue (runner shipped go1.26.8, workflow pins go1.26.7 via go.mod), not a code regression from the recent PR.

#### Suggested remediation

1. Pin the second `Setup Go` step to the same source of truth as the first (`go-version-file: go.mod` instead of the loose `go-version: '1.26'`), so both steps resolve to go1.26.7 and avoid mixed-toolchain build caches.
2. Alternatively, set `GOTOOLCHAIN=local` (already set as an env var in this job) is not sufficient by itself — verify `GOFLAGS`/cache isolation between the two setup-go invocations, or disable caching (`cache: false`) on the first step too if a second, differently-versioned Go install is required later in the same job.
3. Consider consolidating to a single `actions/setup-go` step per job when multiple steps in the same job need Go, to eliminate cross-step toolchain drift entirely.
4. Re-run the `Smoke Copilot - AOAI (Entra)` workflow after the fix to confirm the `aoai-model` deployment succeeds again.

#### Link

- Failing run: https://github.com/github/gh-aw/actions/runs/34545423677

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `api.github.com`
>
> [!TIP]
> `api.github.com` is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding `api.github.com` to `network.allowed`, use `tools.github.mode: gh-proxy` for direct pre-authenticated GitHub CLI access without requiring network access to `api.github.com`:
>
> ```yaml
> tools:
> github:
> mode: gh-proxy
> ```
>
> See [GitHub Tools](https://github.github.com/gh-aw/reference/github-tools/) for more information on `gh-proxy` mode.
>
> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "api.github.com"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

> Generated by [🚨 Deployment Incident Monitor](https://github.com/github/gh-aw/actions/runs/34545763261) · copilot · auto · 49.4 AIC · ⌖ 7.39 AIC · ⊞ 6.4K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Fdeployment-incident-monitor%22&type=issues)
> - [x] expires on Sep 17, 2026, 4:24 PM UTC-08:00

Open .github/workflows/smoke-copilot-aoai-entra.lock.yml and compare the two actions/setup-go steps, starting with their version and cache settings. Make both steps use the same Go version source, then rerun the Smoke Copilot - AOAI (Entra) workflow and confirm the gh-aw CLI build and aoai-model deployment complete successfully.

Written by the indexing model from the issue text.