Problem with MPI-IO's shared file pointer

Nobody has claimed this yet.

@kathrynmohror amd @adammoody , I've observed a problem when testing another of the ROMIO tests. This is a test that addresses the use of shared file pointers. The original program, shared_fp.c, is posted at

https://github.com/pmodels/mpich/blob/master/src/mpi/romio/test/shared_fp.c

In this program, a shared file pointer is used, and each MPI rank writes a buffer of size 4,096 KBytes, using the function MPI_File_write_shared, to a common file. That function implies that the writes are such that the order of serialization is not deterministic.

The integer data buffer written to the file is set with the following scheme, where COUNT is 1,024:

for (i = 0; i < COUNT; i++) buf[i] = COUNT * myrank + i;

I created a reproducer of the problem with the program posted below, which can be used with plain MPI or with UNIFY. With plain MPI, running with 2 processors generates a file of size 8,192 bytes, and running with 4 processors generates a file of size 16,384 bytes. The contents of the file are consistent with the form of generation above, although the order inside the file varies. Running with Unify, I always get a file of size 4,096 bytes, regardless of the number of processors used. So, I suspect that Unify is having trouble to handle the shared file pointer correctly.

Test code:

```
#include "mpi.h"
#include
#include
#include

#ifdef UNIFY
#include
char *filename="ufs:/unifyfs/datafile-u" ;
int ret;
#else
char *filename="datafile-m";
#endif

#define COUNT 1024

/* tests shared file pointer functions */

int main(int argc, char **argv)
{
int *buf, i, myrank, nranks, len, sum, global_sum;
int errs = 0, toterrs, errcode;
MPI_File fh;
MPI_Status status;

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
MPI_Comm_size(MPI_COMM_WORLD, &nranks);
#ifdef UNIFY
ret = unifyfs_mount("/unifyfs", myrank, nranks, 0);
if (ret) {
printf("[%d] unifyfs_mount failed (return = %d)\n", myrank, ret);
MPI_Abort(MPI_COMM_WORLD, 1);
}
#endif

buf = (int *) malloc(COUNT * sizeof(int));

for (i = 0; i < COUNT; i++) buf[i] = COUNT * myrank + i;

errcode = MPI_File_open(MPI_COMM_WORLD, filename,
MPI_MODE_CREATE | MPI_MODE_RDWR, MPI_INFO_NULL, &fh);
if (errcode != MPI_SUCCESS) {
fprintf(stderr, "[%d] Error on MPI_File_open: %d\n",myrank,errcode);
errs++;
}

errcode = MPI_File_write_shared(fh, buf, COUNT, MPI_INT, &status);
if (errcode != MPI_SUCCESS) {
fprintf(stderr, "[%d] Error on MPI_File_write_shared: %d\n",myrank,errcode);
errs++;
}

for (i = 0; i < COUNT; i++) buf[i] = 0;
MPI_Barrier(MPI_COMM_WORLD);

errcode = MPI_File_seek_shared(fh, 0, MPI_SEEK_SET);
if (errcode != MPI_SUCCESS) {
fprintf(stderr, "[%d] Error on MPI_File_seek_shared: %d\n",myrank,errcode);
errs++;
}

errcode = MPI_File_read_shared(fh, buf, COUNT, MPI_INT, &status);
if (errcode != MPI_SUCCESS) {
fprintf(stderr, "[%d] Error on MPI_File_read_shared: %d\n",myrank,errcode);
errs++;
}

MPI_File_close(&fh);

sum = 0;
for (i = 0; i < COUNT; i++) sum += buf[i];

MPI_Allreduce(&sum, &global_sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);

if (global_sum != (((COUNT * nranks - 1) * (COUNT * nranks)) / 2)) {
errs++;
fprintf(stderr, "[%d] Error: sum %d, global_sum %d, %d\n", myrank,
sum, global_sum, (((COUNT * nranks - 1) * (COUNT * nranks)) / 2));
}
free(buf);

MPI_Allreduce(&errs, &toterrs, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
if (myrank == 0) {
if (toterrs > 0) {
fprintf(stderr, "Found %d errors\n", toterrs);
} else {
fprintf(stdout, " No Errors\n");
}
}
#ifdef UNIFY
unifyfs_unmount();
#endif
MPI_Finalize();
return 0;
}
```

Start with the ROMIO test referenced at src/mpi/romio/test/shared_fp.c and run the supplied reproducer with plain MPI and UNIFY at different rank counts. Compare shared-write file sizes and the values read after MPI_File_seek_shared; done means UNIFY preserves all ranks' writes and the validation reports no errors.

Written by the indexing model from the issue text.