Timeout when jobs are waiting
- Dominant language
- Python
- Stars
- 6
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
When I run 4 hw jobs in parallel, half of them end up getting timeout errors. I suspected that this is because generating AFI stage cannot be run in parallel and the waiting time added up to reach the timeout for AFI stage.
In these two, timeout happens in the beginning of AFI stage
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/2sfeH7kJqyg.html
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/w57woT8LUMs.html
However, some of the jobs timeout in the beginning, or middle of make stages
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/wpc3A_L4UDI.html
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/fjMiNTK1b2E/log.txt
And the synthesis timeout is already 20000. I also suspect when jobs are waiting for entering make stage, the waiting time counts towards timeout?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing four parallel hardware jobs and inspect the linked AFI and make-stage logs. Compare queued time with the configured stage timeouts, including the reported synthesis timeout of 20000; done means waiting time is handled correctly and parallel jobs no longer fail unexpectedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100