Step restart may not work with many workers and long run times
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 151
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
With multiple workers and long simulation run times, only half of the workers wake up after the step restart.
I am including the "dummy_simulation.txt" spec and the "workers.txt" batch script. To reproduce the problem on a system with slurm, type:
merlin run dummy_simulation.txt
sbatch workers.txt
and, as soon as the job has started running and the tmp_restart_test_*/dummy*/* subdirectories have been created, type
more tmp_restart_test_*/dummy*/*/*log
and count how many samples reported "Restarting". All 8 samples should restart but only 4 of them do.
Description of the workflow:
-run a "sleep 1" simulation
-restart (after a one-second delay)
-print a line including "Restarting"
-run a "sleep 200" simulation
-terminate: none of the restarted "sleep 200" simulations should finish since the allocation is set to die after 3 minutes.
However, if the "merlin resources" block is removed from the spec, and the test is repeated, all 8 samples report "Restarting", as expected. It may be worth comparing the celery command that is executed in these two scenarios.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached dummy_simulation.txt and workers.txt files, then run merlin run dummy_simulation.txt and sbatch workers.txt on a Slurm system. Inspect the generated logs for Restarting and compare the Celery command in runs with and without the merlin resources block; done means all 8 samples restart as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100