Scavenging the last checkpoint while playing nice with AXL

Nobody has claimed this yet.

The scavenge logic has not really been updated to account for async flushes. In particular, it will be competing with any async transfer that is ongoing and trying to write to the same files. This needs to be updated for AXL in general and BBAPI in particular.

Some context...

First, why is a scavenge useful? For applications that can checkpoint to cache, some might be configured to write all of their checkpoints to cache, and throw away most of them (or even all of them) without transferring them to the parallel file system. In an extreme case, the application might keep all checkpoints in cache and never write one out to the parallel file system until the very end of its allocation.

For those cases, now what happens if a node dies and there is too little time or too few remaining nodes to restart the job in the allocation? In that case, the application will have a checkpoint in cache that can be recovered, and that checkpoint will represent many hours of compute time that we'd prefer not to lose. Even if we lost a node, there may be enough redundancy data to rebuild all files, but we need to copy those checkpoint files and the associated redundancy data to the parallel file system, where we can then rebuild it.

This is the role of the scavenge operation, which optionally executes as a set of scripts at the end of a user's job script.

How does this all work?

During SCR_Complete, if the dataset is valid, SCR records the fact that it has a dataset in ```CACHE``` in the flush file.
https://github.com/LLNL/scr/blob/357855308d0f4d7de60993a68e31796a2902f685/src/scr.c#L1534

The flush file is kept in ```${SCR_PREFIX}.scr/flush.scr```. A flush is then started if the dataset needs to be flushed (any output dataset or select checkpoints). This will either be an async flush or sync flush depending on configuration.

If a flush is started, the dataset is also marked as ```FLUSHING``` in the flush file.
https://github.com/LLNL/scr/blob/357855308d0f4d7de60993a68e31796a2902f685/src/scr_flush_sync.c#L153
https://github.com/LLNL/scr/blob/357855308d0f4d7de60993a68e31796a2902f685/src/scr_flush_async.c#L119

SCR checks for the completion of any outstanding async flush at various points in time. When a flush completes, whether a sync or async flush, SCR then updates the flush file to mark the dataset as also being on the ```PFS```.
https://github.com/LLNL/scr/blob/357855308d0f4d7de60993a68e31796a2902f685/src/scr_flush.c#L401

If the job finishes on its own and calls SCR_Finalize, then during SCR_Finalize, SCR checks whether that is an outstanding async flush. If so, it can either wait on it to complete or kill it. If it kills the flush, it starts a sync flush, which is assumed to be faster.
https://github.com/LLNL/scr/blob/357855308d0f4d7de60993a68e31796a2902f685/src/scr.c#L2158

If the scr run scripts are being used, ```scr_postrun``` will check the flush file to see if there are any datasets in ```CACHE``` that are not on the ```PFS```. It checks that *all* output datasets are on the ```PFS``` and it also checks that the most recent checkpoint is on the ```PFS```.
https://github.com/LLNL/scr/blob/357855308d0f4d7de60993a68e31796a2902f685/scripts/common/scr_postrun.in#L116

If needed, ```scr_postrun``` executes an ```scr_scavenge --id ``` for each dataset it needs to collect. The ```scr_scavenge``` script in turn launches an ```scr_copy --id ``` executable on each compute node (via pdsh).

The ```scr_copy``` executable synchronously copies all files for the given dataset id from cache to the parallel file system. The user data files go to their original path, and scr_copy creates directories as needed. Since we are running the scavenge logic, we assume that something bad has happened and so scr_copy also copies the redundancy data files. Those files go into the corresponding scr dataset directory ```${SCR_PREFIX}/.scr/scr.dataset./``` along with a "filemap" for each rank, which records which MPI rank owns which file.

Once all scr_copy finishes on all nodes, scr_postrun executes an ```scr_index --add ``` command. The ```scr_index --add``` command inspects the named dataset. It reads the filemap files and then checks that it has all user data files. If not, it attempts to rebuild any missing files using the redundancy data files. The rebuild logic is encapsulated in the ```scr_rebuild_partner``` or ```scr_rebuild_xor``` executables. If all user data files can be recovered, ```scr_index --add``` marks the dataset `````` as being valid in the scr index file ```${SCR_PREFIX}/.scr/index.scr```.

This entire process is repeated for each dataset that can be scavenged, i.e., all output sets and the most recent checkpoint, if they are not already on the file system.

A few things that come to mind:
- scr_copy could be updated to use AXL instead of copying the files itself.
- scr_copy must at least coordinate with AXL so that it does not compete with it, e.g., cancel AXL if scr_copy will also transfer files or have scr_copy wait on the existing transfer
- If we're trying to scavenge, we typically want to copy files as fast as possible, since we are probably racing the clock to beat the allocation time limit. Note that this might be counter to an ongoing background transfer, which might be throttling the write on purpose (like BBAPI).
- Having said that, a different approach we can use with the BBAPI is to have the scavenge logic just kick of BBAPI transfers and exit the allocation. Then the scr_index/rebuild logic must be executed as part of the bsub post-stage script where we wait on those transfers to complete.

Start by tracing scr_postrun, scr_scavenge, and scr_copy alongside the async paths in src/scr_flush_async.c and the completion handling in src/scr_flush.c. Determine how AXL and BBAPI transfers should coordinate with scavenging, then verify that the selected datasets are safely copied or transferred, indexed, and rebuilt before postrun completes.

Written by the indexing model from the issue text.