[Hosted agents 11] Validate private networking, image build, and version rollback
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 321
- Avg merge
- 6h 22m
- Merged PRs (30d)
- 26
Description
## Summary
Validate hosted-agent deployment under the supported private-network topology, including image build/pull and immutable-version rollback.
## Why will we implement this?
- **Problem / opportunity:** Public GitHub-hosted runners cannot prove access to a private ACR/private endpoints, and hosted-agent private ACR support has project-date constraints.
- **Business value / outcome:** Network-isolated deployments have a documented, tested path rather than an unsafe public-access workaround.
- **Success metrics (how we know it worked):** The selected VNet-connected runner, jumpbox, or ACR Task route builds and deploys successfully; private endpoints remain closed; rollback selects a previous hosted version.
## What does it do? (Functional Overview)
- **Core behavior:** Exercise the task-01 private build decision, ACR/project eligibility, private DNS/endpoints, hosted image pull, egress, and version reversal.
- **Data collection / storage needs:** No credentials in logs; temporary images/resources follow cleanup policy.
- **Data analysis / reporting needs:** Record topology and failure/recovery evidence without private environment names.
- **Nice to have (stretch goals):** Reusable private validation workflow on an approved runner.
## Components
- **Components (check all that apply):**
- [x] gpt-rag-orchestrator
- [ ] gpt-rag-ingestion
- [ ] gpt-rag-ui
- [ ] gpt-rag-mcp
## Dependencies and acceptance criteria
- **Depends on:** none. #592 was closed on 2026-09-03; the validation remaining here is self-contained.
- No public ACR access is enabled as a workaround.
- Private route and immutable-version rollback both succeed.
- If evidence cannot be produced, network-isolated hosted mode remains explicitly unsupported.
## Remaining scope at the currently published pins
Superseding the 2026-08-11 and 2026-08-12 comments, whose pull-request and pin
statements have since gone stale. Umbrella `v3.8.3` pins `gpt-rag-ui v2.6.2`,
`gpt-rag-orchestrator v4.1.1`, `gpt-rag-ingestion v2.7.3` and AI Landing Zone
`v2.5.1`.
**One item remains.**
- [ ] Exercise digest-to-digest rollback on a live network-isolated deployment.
The 2026-08-07 pass created a replacement hosted version from the *same*
immutable digest and routed traffic back to the previous version. That
proves hosted-version reversal, but it does not prove serving a *different,
earlier* digest, which is what the ADR-0001 fitness function requires. The
mechanism itself is implemented and unit-tested — `--image-version
sha256:` in `config/deployment/hosted_prepare.py` skips the build,
reuses the supplied digest, and clears generated-image provenance so the
coherence guard in `config/deployment/composition.py` does not fire. What
is missing is the live execution.
### Resolved: VNet-injected ACR Tasks agent pool
Previously listed here as outstanding on the premise that
`deployAcrTaskAgentPool` stays at its `main.parameters.json` default of `false`,
leaving a fresh network-isolated deployment with no dedicated pool. That premise
does not hold on the hosted path, which is this issue's scope.
- `config/deployment/composition.py` forces `deployAcrTaskAgentPool` to `true`
whenever the deployment is hosted and `NETWORK_ISOLATION` is truthy, so the
operator default cannot leave the pool unrequested. The `false` default is
preserved deliberately for classic mode and is covered by
`test_classic_composition_preserves_acr_pool_operator_setting`.
- `config/deployment/hosted_prepare.py` additionally refuses to prepare a
network-isolated hosted build when `ACR_TASK_AGENT_POOL` is absent, so the
fallback to a publicly-egressing shared pool is fail-closed rather than silent.
- The pool's own bootstrap defect — this issue's historical "Defect C" — was
fixed in AI Landing Zone `v2.4.1` and confirmed live in a disposable
network-isolated deployment, recorded in that repository's `CHANGELOG.md`
against this issue.
- `v2.5.0` left the private ACR, VNet-injected pool, firewall, private endpoint
and DNS topology under their existing flags, and `v2.5.1` states that beyond
the `agent365.svc.cloud.microsoft` FQDN "no other new firewall rule, resource,
or feature flag is introduced". The validated build path is therefore
unchanged between `v2.4.1` and the currently pinned `v2.5.1`.
The 2026-08-12 failure reported in this thread was a **classic-mode** run whose
environment left the flag at `false`. That failure mode is not reachable on the
hosted path.
### Resolved: rollback documentation
ADR-0001 requires rollback by version reversal to be documented. Added to
`docs/hosted_agent_release_matrix.md` by #682, merged on 2026-09-04 and
published to the docs site.
### Not tracked here
The 2026-08-07 pass surfaced a separate platform limitation: managed Conversation
items stayed readable through the project API after compute replacement, but the
hosted Responses endpoint returned HTTP 404 when reusing that Conversation, on
both the replacement version and the original version after rollback. A fresh
invocation on the rolled-back version succeeded. That is conversation continuity,
not private networking or image build, and is tracked on #611.
The `agent365.svc.cloud.microsoft` firewall rule is **no longer outstanding**. It
shipped in AI Landing Zone `v2.5.1` (2026-08-11), the version currently pinned, as
an FQDN on the existing `AllowContainerAppsPlatform` application rule in
`modules/networking/azure-firewall.bicep`, with `Test-FirewallAgent365ObservabilityContract.ps1`
guarding it. A fresh deployment at the current pin inherits it.
Contributor guide
Research direction
Read ADR-0001 and the remaining-scope section, then inspect config/deployment/hosted_prepare.py and config/deployment/composition.py, especially the --image-version sha256: path. Use an approved network-isolated hosted deployment with an earlier digest and record private-route success, rollback evidence, credential-safe logs, and cleanup; if it cannot be demonstrated, document hosted mode as unsupported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cloud, devops, infrastructure, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100