Problem with strided writes by multiple ranks

Nobody has claimed this yet.

As I have promised to @kathrynmohror during the Unify call yesterday, I'm opening this issue to report a problem that I observed in one of the ROMIO tests. Opening this issue also allows @adammoody to check and comment about the problem.

I first observed the problem when porting to Unify one of ROMIO's tests (noncontig.c), which is posted at

https://github.com/pmodels/mpich/blob/master/src/mpi/romio/test/noncontig.c

That program is in fact composed of three tests. The second test works right with Unify, but the first and third produce wrong results. The first and third tests write to the file in an interleaved fashion, hence I created a reproducer of the third test, which I'm posting here. This test must be run with exactly two processors. The test creates a new datatype with a stride, and then both ranks write to the same file, such that their data will be interleaved in the file. This is how each rank defines the data to be written:

buf[i] = i + myrank * 5000 ;

Running WITHOUT Unify (i.e. plain MPI), the resulting file looks like this:

(0) 0
(1) 5000
(2) 1
(3) 5001
(4) 2
(5) 5002
. . .

However, running under Unify, I obtain a resulting file with contents like this:

(0) 0
(1) 5000
(2) -1
(3) 5001
(4) -1
(5) 5002
(6) -1
(7) 5003
. . .

If I change the code to make just one of the processors to write to the file, they write correctly their part (either rank=0 or rank=1). Thus, writing with a stride by one of the processors seems to work fine. But having the two ranks writing, and forcing them to write in a certain order, does NOT change the wrong result in the file as shown above. So, the problem seems to arise when both ranks try to write to the file in the interleaved fashion.

This is the test code:

```
#include "mpi.h"
#include
#include
#include
#ifdef UNIFY
#include
char *filename="ufs:/unifyfs/datafile-u" ;
#else
char *filename="datafile-m";
#endif

/* tests noncontiguous reads/writes using independent I/O */

#define SIZE 5000

#define VERBOSE 1

int main(int argc, char **argv)
{
int *buf, i, myrank, nranks, b[3], ret;
MPI_Aint d[3];
MPI_File fh;
MPI_Status status;
MPI_Datatype typevec, newtype, t[3];
MPI_Info info;

MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nranks);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
#ifdef UNIFY
ret = unifyfs_mount("/unifyfs", myrank, nranks, 0);
if (ret) {
printf("[%d] unifyfs_mount failed (return = %d)\n", myrank, ret);
MPI_Abort(MPI_COMM_WORLD, 1);
}
#endif

if (nranks != 2) {
fprintf(stderr, "Run this program on two processes\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}

buf = (int *) malloc(SIZE * sizeof(int));

MPI_Type_vector(SIZE / 2, 1, 2, MPI_INT, &typevec);

/* create a struct type with explicitly set LB and UB; displacements
* of typevec are such that the types for the two processes won't
* overlap.
*/
b[0] = b[1] = b[2] = 1;
d[0] = 0;
d[1] = myrank * sizeof(int);
d[2] = SIZE * sizeof(int);
t[0] = MPI_LB;
t[1] = typevec;
t[2] = MPI_UB;

/* keep the struct, ditch the vector */
MPI_Type_struct(3, b, d, t, &newtype);
MPI_Type_commit(&newtype);
MPI_Type_free(&typevec);

MPI_Info_create(&info);
/* I am setting these info values for testing purposes only. It is
* better to use the default values in practice. */
#if 0
MPI_Info_set(info, "ind_rd_buffer_size", "1209");
MPI_Info_set(info, "ind_wr_buffer_size", "1107");
#endif
#if VERBOSE
fprintf(stderr,
"\ntesting contiguous in memory, noncontiguous in file using independent I/O\n");
#endif
MPI_Barrier(MPI_COMM_WORLD);

MPI_File_open(MPI_COMM_WORLD, filename, MPI_MODE_CREATE | MPI_MODE_RDWR, info, &fh);

/* set the file view so that we have interleaved access again */
MPI_File_set_view(fh, 0, MPI_INT, newtype, "native", info);

/* this time write a contiguous buffer */
for (i = 0; i < SIZE; i++) buf[i] = i + myrank * SIZE ;

MPI_File_write(fh, buf, SIZE, MPI_INT, &status);

MPI_Barrier(MPI_COMM_WORLD);
sleep(2);

/* fill buffer with -99's; this time they will all be overwritten */
for (i = 0; i < SIZE; i++) buf[i] = -99;
MPI_File_read_at(fh, 0, buf, SIZE, MPI_INT, &status);
fprintf(stderr,"[%d] BUF READ: %d %d %d %d %d %d\n",myrank,buf[0],buf[1],buf[2],buf[3],buf[4],buf[5]);

MPI_File_close(&fh);
MPI_Barrier(MPI_COMM_WORLD);

MPI_Type_free(&newtype);
MPI_Info_free(&info);
free(buf);
#ifdef UNIFY
unifyfs_unmount();
#endif
MPI_Finalize();
return 0;
}

```

Start with src/mpi/romio/test/noncontig.c and the reproducer included in the issue, then trace MPI_File_set_view, MPI_File_write, and MPI_File_read_at under UnifyFS. Run the reproducer with exactly two processes and compare its interleaved output with plain MPI; done means both ranks' values are written and read correctly under UnifyFS.

Written by the indexing model from the issue text.