Problem with MPI_File_preallocate

Nobody has claimed this yet.

As had been requested by @adammoody I created a test to document a problem with MPI_File_preallocate under Unify. This function is used in one of the ROMIO examples, misc.c available at

https://github.com/pmodels/mpich/blob/master/src/mpi/romio/test/misc.c.in

However, to make the problem more explicit, I created a reproducer program, shown below. This program follows this sequence of MPI-IO calls (only major functions shown):

MPI_File_open
MPI_File_preallocate (size=8KB)
MPI_File_write_at (total=8KB)
MPI_File_close

After each of the first three calls, each rank checks the file size with 'MPI_File_get_size'. There are barriers between the various phases just to make sure the ranks are always in sync within the phases.

Running the program with 4 processors, in a SINGLE node of Quartz under Unify I obtain these sizes:

[3] After File_open: SIZE=0
[2] After File_open: SIZE=0
[0] After File_open: SIZE=0
[1] After File_open: SIZE=0
[3] After preallocate(8KB): SIZE=0
[2] After preallocate(8KB): SIZE=0
[1] After preallocate(8KB): SIZE=0
[0] After preallocate(8KB): SIZE=8192
[0] After 8KB write_at: SIZE=4096
[1] After 8KB write_at: SIZE=6144
[2] After 8KB write_at: SIZE=8192
[3] After 8KB write_at: SIZE=8192

Thus, it seems that only Rank=0 finds the proper file size after the preallocate. And after all ranks write to the file, the ranks also have trouble in obtaining the resulting size.

Meanwhile, running WITHOUT Unify, also with 4 processors, I get the expected values, as follows:

[0] After File_open: SIZE=0
[1] After File_open: SIZE=0
[3] After File_open: SIZE=0
[2] After File_open: SIZE=0
[2] After preallocate(8KB): SIZE=8192
[0] After preallocate(8KB): SIZE=8192
[1] After preallocate(8KB): SIZE=8192
[3] After preallocate(8KB): SIZE=8192
[0] After 8KB write_at: SIZE=8192
[1] After 8KB write_at: SIZE=8192
[3] After 8KB write_at: SIZE=8192
[2] After 8KB write_at: SIZE=8192

The contents of the 8KB files produced with and without Unify are the same.

This is the reproducer test code:

```
#include "mpi.h"
#include
#include
#include
#define BUFLEN 2048

int main(int argc, char **argv)
{
int buf[BUFLEN], myrank, i, nints, nranks, errcode=0;
MPI_File fh;
MPI_Status status;
MPI_Offset offset,size;

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
MPI_Comm_size(MPI_COMM_WORLD, &nranks);
errcode = unifyfs_mount("/unifyfs", myrank, nranks, 0);
if (errcode) {
printf("[%d] unifyfs_mount failed (return = %d)\n", myrank, errcode);
exit(-1);
}
for (i=0; i

Start with the reproducer in the issue and the referenced src/mpi/romio/test/misc.c.in test, running it with and without UnifyFS. Compare MPI_File_get_size after MPI_File_preallocate and MPI_File_write_at; done means the reported sizes are consistent and match the expected 8192-byte results under UnifyFS.

Written by the indexing model from the issue text.