Account for BBAPI post-stage in SCR

Using its post-stage functionality, the BBAPI will continue to transfer files in the background even after an application ends. LSF allows the user to register poststage scripts in the bsub command to detect and act on the status of post-stage transfers. To allow for BBAPI transfers at the end of an SCR job, we need to write some logic to finalize those transfers. There are numerous items to address.

In AXL we'd need to effectively wait/resume on each of those in order to finish its transfers, e.g., rename files from temporary to final file name and set metadata on final files https://github.com/ECP-VeloC/AXL/issues/75

In SCR, we'd need to update the SCR index file to mark the transfer as good or bad. Assuming that most transfers succeed, it would be nice to wait and mark the checkpoint as valid, so that any subsequent run could then restart from that checkpoint. However, that also introduces a race condition in which job 1 flushes a checkpoint in post-stage, but job 2 starts up before that transfer has completed. In that case, job 2 would restart from an older checkpoint, and then it would likely rewrite and reflush the same checkpoint as job 1, perhaps while the system is still busy flushing job 1's checkpoint. At that point, we'd have two different jobs trying to write the same checkpoint, and that's going to break things.

We could update the SCR scavenge logic to use AXL/BBAPI to start a transfer and exit the job allocation instead of synchronously copying files to the file system at the end of the allocation. We'd need a post-stage script that both waits on the transfer to finish *and* executes any rebuild logic, as our scavenge normally does.

We need to modify the existing scavenge logic to detect and deal with async transfers in case the dataset it's trying to scavenge is the one being transferred. We'd want to at least add the redundancy files. That could be done as a separate transfer, or we could cancel the first and restart it after adding the redundancy files.

As a short term fix, we can modify SCR to avoid starting any async transfers when the job gets close to the end of its allocation time limit. All transfers would switch to use synchronous mode.