Problem in Unify/MPI-IO reading data across a "boundary" between two previously written ranges in a file

Nobody has claimed this yet.

@adammoody , @kathrynmohror : After doing extensive testing with one of the failing parallel HDF5 codes from their testsuite (cchunk5), I think I finally understand why it is failing, although I haven't looked yet at how to fix it in Unify. I used Recorder (without Unify) to see exactly what that HDF5 program was doing for I/O, and then, correlating the traces to the logs from a Unify-based execution, I can see where it goes wrong with Unify (my Unify version is based on Mike's new margotree branch). Based on that, I created this reproducer program below, which only uses MPI-IO (i.e. no HDF5), as I think this will make it easier to look for a fix.

The example below must be run in 4 processors. Running without Unify, file "datafile-m" will be created, with 400 bytes. Running under Unify, file "datafile-u", which also has 400 bytes, will be created. A byte-by-byte comparison between those two files will show differences between bytes 101:107 across the two files. This is an error similar to what I saw with that HDF5 test.

In this code, each rank initially writes 100 bytes to disjoint locations in the file. Then rank-0 tries to read 12 bytes starting at offset=95. This would have to read across two ranges, one range [0:99] written previously by rank-0, and the range [100:199] previously written by rank-1. The read is done with function MPI_File_read_at.

After the MPI_File_read_at call, the number of bytes actually read is retrieved in status.count_lo (according to the MPI_Status structure of MPI). Without Unify, that number is 12. However, with Unify, that number is *5* (i.e. the amount of bytes between 95 and 99). Those values (12 and 5) are printed by the executions of the program in stdout.

This is the test code (to be compiled for Unify, it must use -DUNIFY):

```
#include "mpi.h"
#include
#include
#ifdef UNIFY
#include
char *filename="ufs:/unifyfs/datafile-u" ;
#else
char *filename="datafile-m";
#endif

#define SIZE 100

int main(int argc, char **argv)
{
int i, myrank, nranks, ret, count, nread;
char buf[SIZE];
MPI_File fh;
MPI_Status status;
MPI_Info info=MPI_INFO_NULL;
MPI_Offset offset;

MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nranks);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
#ifdef UNIFY
ret = unifyfs_mount("/unifyfs", myrank, nranks, 0);
if (ret) {
printf("[%d] unifyfs_mount failed (return = %d)\n", myrank, ret);
MPI_Abort(MPI_COMM_WORLD, 1);
}
#endif

if (nranks != 4) {
fprintf(stderr, "Run this program on 4 processes\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}

for (i=0; i

Compile the supplied MPI-IO reproducer with -DUNIFY and run it on 4 processes, comparing datafile-m with datafile-u and the reported number_read. Trace unifyfs_gfid_read_reqs(), process_read_data() in unifyfs.c, and invoke_client_read_rpc() in margo_client.c for the reads at offsets 95 and 100. Done means the boundary-spanning read reports 12 bytes and the generated files match.

Written by the indexing model from the issue text.