Merlin needs blocking parallel launches

Nobody has claimed this yet.

The celery tasks in the merlin package require blocking parallel launches to be substituted for $(LAUNCHER) in the step scripts. This is due to the way celery handles task completions. If a non-blocking parallel launch is used, a poll method would need to be added to check for the job completion so the task can mark itself completed, increasing complexity. The current merlin implementation for LAUNCHER (see PR below) is to derive from the SlurmScriptAdapter class and substitute the blocking commands for flux and LSF. There may be a better way to do this on the maestro side.

The issue is here:
LLNL/merlin#59

First PR for implementation is here:
LLNL/merlin#72

We have also found that extra options are required in some cases. These options are scheduler specific, so maybe a scheduler arg can be added to the run section.

run:
cmd: ${LAUNCHER) prog args
nodes: 4
tasks: 64
slurm: "--exclusive --mpibind=off --signal=USR1@32000"

The slurm section would only be added to the LAUNCHER string if the batch type is slurm.

Added in LLNL/merlin#138

Start by reviewing the current LAUNCHER implementation, the SlurmScriptAdapter approach, and the related Merlin issues and pull requests (#59, #72, and #138). Determine the agreed Maestro-side design for blocking launches and scheduler-specific options; done means the required launch behavior and options work for the affected scheduler types without requiring an added polling path.

Written by the indexing model from the issue text.