Try 1ES pools for Windows builds to remove need for retry logic
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 431
- Forks
- 44
- Avg merge
- 21h 18m
- Merged PRs (30d)
- 30
Description
For this flakiness issue, we're adding retries to the Windows jobs:
The only known explanations for the flakiness are Windows antivirus or an Azure scan gone wrong and keeping the file open.
https://teams.microsoft.com/l/message/19:afba3d1545dd45d7b79f34c1821f6055@thread.skype/1638482594255?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&parentMessageId=1638482594255&teamName=.NET%20Core%20Eng%20Services%20Partners&channelName=First%20Responders&createdTime=1638482594255
You can never, really, fully shut off defender or WU, but we do our best to
One thing we just noticed is that Azure Security pack stuff seems to be "turning on" for our VMs even when we specify the properties not to
In the end trying to figure it out is madness, Retry Is The Way.
We've been using Microsoft-hosted pools. We could try out 1ES pools, where:
- The environment is different.
- The hardware is (last I heard) more powerful.
- dnceng is more directly aware of the attempts made to disable scans. (The Microsoft-hosted agents do seem to attempt to do this, but we/dnceng only saw this by reading the scripts--not involved in writing them.)
- Any change, big or small, could make this work more reliably. We don't have a lot of info--not even sure about the cause.
- E.g. a faster disk could make scans complete in time to avoid breaking our builds?
- We are more likely to have success with a 1ES pool bug report/fix, because we have more direct lines of communication.
After switching to the 1ES pool, we can use https://github.com/jaredpar/runfo to scan the pipeline logs for retries and see if the number goes down. (Look for Running 'make' attempt 2 of 5....)
We can also run stress test jobs like https://dev.azure.com/dnceng/internal/_build/results?buildId=1495624&view=results to get more data, quickly.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the Windows build pool configuration and the referenced stress-test job. Switch the Windows jobs to a 1ES pool, then use runfo to scan pipeline logs for messages such as “Running 'make' attempt 2 of 5...”. Done means comparing retry frequency with the Microsoft-hosted pool and documenting whether reliability improves.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go
- Domain
- build-system, ci-cd, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100