cucapra / cucapra/polyphemus

Timeout when jobs are waiting

Open
#8 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
2
PR merge metrics
No merged PRs in 30d

Description

When I run 4 hw jobs in parallel, half of them end up getting timeout errors. I suspected that this is because generating AFI stage cannot be run in parallel and the waiting time added up to reach the timeout for AFI stage.
In these two, timeout happens in the beginning of AFI stage
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/2sfeH7kJqyg.html
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/w57woT8LUMs.html
However, some of the jobs timeout in the beginning, or middle of make stages
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/wpc3A_L4UDI.html
http://ec2-54-234-195-6.compute-1.amazonaws.com:5000/jobs/fjMiNTK1b2E/log.txt
And the synthesis timeout is already 20000. I also suspect when jobs are waiting for entering make stage, the waiting time counts towards timeout?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing four parallel hardware jobs and inspect the linked AFI and make-stage logs. Compare queued time with the configured stage timeouts, including the reported synthesis timeout of 20000; done means waiting time is handled correctly and parallel jobs no longer fail unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.