Release automation shouldn't reserve CI machines while polling

The release automation infra contains a lot of polling steps: poll until an upstream tag is available, poll until a PR is merged, poll until an AzDO build is done, .... Since we poll in an AzDO pipeline build, this uses up a build machine in the pool for as long as polling continues while doing essentially no productive work. Our builds haven't been identified as a particular problem for build pool throughput, but it is not a good approach.

A few mitigations/solutions we have in mind if this turns out to be a problem:

• (Bad) Add "user approval" steps on a server-side job to let the pipeline stop without using up a VM.
  • This means we don't keep a machine busy polling. However, it puts a burden on the dev running the release.
  • We currently have one of these to prevent the go-images automation from starting to poll bizarrely early. But we will probably want to remove it to streamline the release process.
  • Any solution that involves "user approval" steps has significant overhead: when the next step starts running that needs a real agent, it has to reserve a machine, so when we repeatedly interrupt the build, this delay gets multiplied.
• (Better) Move the polling to an external system that doesn't rely on an AzDO build pool.
  • We can have the pipeline call out to Azure Functions (potentially Durable Functions) to have it start polling and report back later.
  • This could be implemented using an agentless/server job, with:
    • AzureFunction@1 - Kick off the polling function.
    • ManualValidation@0 - Receive the callback in the form of a service account pressing the approve button.
• (Incredible) Add or discover some Azure Pipelines functionality that enables some way to poll without a constantly reserved agent.
• (Infeasible) Add hooks into every service we interact with so that when an action happens (an upstream push, CI completion, AzDO job completion), our service is notified, and so we don't need to do any polling.
  • Making this work for every service doesn't seem feasible, but we could potentially make incremental improvements that use hooks for certain actions like our CI and AzDO build completion.

Start by locating the release automation polling steps that run in Azure DevOps pipeline builds and identify which ones reserve an agent while waiting. Compare the proposed Azure Functions and agentless/server-job approaches, including AzureFunction@1 and ManualValidation@0. Done means polling no longer keeps a CI machine reserved, but the issue does not select a specific implementation.