llnl / llnl/merlin

Step restart may not work with many workers and long run times

Open
#418 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
151
Forks
30
PR merge metrics
No merged PRs in 30d

Description

With multiple workers and long simulation run times, only half of the workers wake up after the step restart.

I am including the "dummy_simulation.txt" spec and the "workers.txt" batch script. To reproduce the problem on a system with slurm, type:

merlin run dummy_simulation.txt
sbatch workers.txt

and, as soon as the job has started running and the tmp_restart_test_*/dummy*/* subdirectories have been created, type

more tmp_restart_test_*/dummy*/*/*log

and count how many samples reported "Restarting". All 8 samples should restart but only 4 of them do.

Description of the workflow:
-run a "sleep 1" simulation
-restart (after a one-second delay)
-print a line including "Restarting"
-run a "sleep 200" simulation
-terminate: none of the restarted "sleep 200" simulations should finish since the allocation is set to die after 3 minutes.

However, if the "merlin resources" block is removed from the spec, and the test is repeated, all 8 samples report "Restarting", as expected. It may be worth comparing the celery command that is executed in these two scenarios.

dummy_simulation.txt
workers.txt

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached dummy_simulation.txt and workers.txt files, then run merlin run dummy_simulation.txt and sbatch workers.txt on a Slurm system. Inspect the generated logs for Restarting and compare the Celery command in runs with and without the merlin resources block; done means all 8 samples restart as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.