Give builders two timeouts: one for expected time, another to see if it would have succeeded if we have given it a higher timeout
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 431
- Forks
- 44
- Avg merge
- 21h 18m
- Merged PRs (30d)
- 30
Description
This is a strategy I've seen used for .NET tests:
- Fail the build after an expected maximum time it should take to run the tests on the hardware. Keeping this relatively tight prevents PRs from unknowingly massively regressing the time.
- Keep the build running until a significantly longer timeout expires. The build still fails, but this way, we get to see if the build would succeed if we were to extend the timeout.
- This is very useful when we do expect a performance regression, or when we don't have time to fix it yet. We can see how much longer we need to extend the timeout for, and whether there's a real hang.
The long tests are long. I just had a build where one longtest succeeded at 53 minutes and another failed due to a 1hr timeout--this strategy would have been useful.
This can be done by giving the AzDO job the long timeout, and make eng/run.ps1 run-builder keep track of the shorter timeout.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading eng/run.ps1, especially the run-builder path, and inspect how the AzDO job currently applies its timeout. Define the shorter expected-runtime failure separately from the longer build-running timeout. Done means a build reports an early timeout while continuing until the longer limit, distinguishing a slow success from a hang.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, powershell
- Domain
- build-system, ci-cd, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100