Support file broadcasting during restart

A common pattern in some codes is to have rank 0 write a root file and then every process, rank 0 included, each writes a per-process checkpoint file. The root file often contains global information that all processes know, but only one rank has to write.

Under normal SCR semantics, the rank that writes the root file is the rank that owns it, and only the owner rank can read the file during a restart. In particular, the file might only exist in node-local storage where the owner rank is running.

The "proper" way to handle such a file is to have the owner rank read the file and broadcast its data to all other ranks via MPI. However, a number of applications are written such that all ranks directly read the root file. They expect the file to be in global storage, and writing the code to broadcast the data is extra work and perhaps not trivial.

To support such codes, the only option right now is to run with SCR_GLOBAL_RESTART=1 so that SCR rebuilds and flushes the current checkpoint to global storage during SCR_Init. In this case, the application should read its checkpoint without using the SCR restart API. This fallback works, but it's a shame that the application can't use the SCR restart API. Also, the root file is typically small, so it's inefficient to have to flush so much data just to deal with one small file.

An alternative would be to add a new SCR_Bcast_file API. This could be used during restart to replicate a file as needed or flush the single file to global storage where all processes could read it. The extra copy could be deleted during the SCR_Complete_restart call. If flushing to the PFS, the file could be placed within the hidden .scr directory for its corresponding dataset. A list of broadcasted files could be built up during restart to later be deleted. The SCR_Route_file logic would need to consider potential broadcasted files.

SCR_Start_restart()

// collective call to replicate "foo" or copy it to PFS
// record in bcasted list
SCR_Bcast_file("foo")

// this proc can now read "foo" even though it may not be owner
SCR_Route_file("foo", newfoo)

// copies of any bcasted files get deleted
SCR_Complete_restart()

Review SCR_Start_restart, SCR_Route_file, and SCR_Complete_restart alongside SCR_GLOBAL_RESTART behavior. Define the collective SCR_Bcast_file lifecycle, including whether it replicates locally or copies to PFS, records temporary copies, and lets non-owner ranks route the file. Done means restart clients can read the broadcast file through SCR and cleanup removes copies at completion.