python: pdsh -Rexec and clustershell ExecWorker

Nobody has claimed this yet.

The -Rexec option tells pdsh to run the given command locally after making some optional substitutions, like replacing %h with a hostname. If one does not specify -Rexec, pdsh falls back to use its default module. The default depends on the configuration of pdsh in general.

https://linux.die.net/man/1/pdsh

ClusterShell seems to be very similar. When given a list of nodes, the "Task" class spawns up a "Worker" class for each node.

https://clustershell.readthedocs.io/en/latest/api/Task.html

There are multiple Worker implementations, including ssh, rsh, pdsh, and "exec". I think it picks WorkerSsh by default. There is also an "ExecWorker" that seems to be similar to the -Rexec option for pdsh, where it runs a command locally and makes substitutions:

https://clustershell.readthedocs.io/en/latest/api/workers/ExecWorker.html

I think we had to resort to -Rexec on systems where users were not allowed to ssh or rsh to their compute nodes. Instead, we had to plug in an srun/aprun job that targeted a specific host to accomplish the same task.

I'm getting the feeling that we'll have to continue to support this kind of -Rexec/srun thing to be portable, since there are likely still HPC sites that opt to not give their users rsh/ssh access.

From our high level logic for scavenge and node_check, I'd like to support a function like parallel_exec(argv=[args], nodes=[nodelist]) that just takes the command someone wants to run and the list of target nodes. The caller shouldn't have to care about *how* that command actually runs on each node. Perhaps the caller could pass in an object that defines how it happens, but the caller could just create that object directly based on SCR config settings or something.

default method for sites where pdsh "just works"
pdsh -w nodelist args

for SLURM systems where one cannot use ssh/rsh
pdsh -Rexec -w nodelist srun -n1 -N1 -w %h args

for APRUN systems where one cannot use ssh/rsh
pdsh -Rexec -w nodelist aprun ...

ClusterShell will have a similar set of options to pick from.

It's still not clear to me how we should handle the difference in selecting the proper option. For sites that allow users to SSH, I like the cleaner option of avoiding the "exec" mess. It seems like it has less that can go wrong. It also avoids creating a bunch of extra SLURM job steps that the user won't expect to see. For sites that prohibit users from direct ssh/rsh, then maybe we can construct the right exec command if we just know the resource manager? Or maybe we'll need the user to select one from a set.

Anyway, this feels like we'll need some new configuration option for SCR (ssh/rsh/pdsh/exec+srun/exec+aprun/etc).

Start by tracing the high-level scavenge and node_check logic and SCR configuration mentioned in the issue; no files or tests are named. Compare the pdsh and ClusterShell execution options described, then define the supported selection behavior and how completion would be verified for SSH/RSH, srun, and aprun configurations.

Written by the indexing model from the issue text.