Problem to handle 4GB file

Nobody has claimed this yet.

@kathrynmohror , @adammoody : As I reported in the call today, I'm having trouble with one of the Romio tests, large_file.c, thus I'm opening this issue to record the problem. That original code is available from

https://github.com/pmodels/mpich/blob/master/src/mpi/romio/test/large_file.c.in

This code works fine with one processor under plain MPI. It also works under Unify if the file has only 2GB, but fails at 4GB. I'm not sure that Unify can indeed handle files of size 4GB or larger, would any of you know?

The code has the following structure:

for (i=0; i<128; i++) {
. . .
MPI_File_write(fh,...); /* write 32 MB */
}
MPI_File_get_size(fh,&size);

The "writes" seem to work fine, but after the records for those writes (completing 4GB) I see the following in the client log:

@ unifyfs_intercept_fd() [unifyfs.c:344] Changing fd from exposed 1024 to internal 0
@ invoke_client_sync_rpc() [margo_client.c:561] invoking the sync rpc function in client
@ invoke_client_sync_rpc() [margo_client.c:570] Got response ret=12
@ unifyfs_sync() [unifyfs-fixed.c:206] failed to flush write index to server for gfid=1953131809

Hence, the writes seem to work fine, but the MPI_File_get_size operation fails.

This is the test code:

```
#include "mpi.h"
#include
#include
#include
#ifdef UNIFY
#include
char *filename="ufs:/unifyfs/datafile-u" ;
int ret;
#else
char *filename="datafile-m";
#endif

/* writes a file of size 4 Gbytes and reads it back.
should be run on one process only*/

#define SIZE 1048576*4 /* no. of long longs in each write/read */
#define NTIMES 128 /* no. of writes/reads */

int main(int argc, char **argv)
{
MPI_File fh;
MPI_Status status;
MPI_Offset size;
long long *buf, i;
int j, myrank, nranks, len, flag, err;

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
MPI_Comm_size(MPI_COMM_WORLD, &nranks);
#ifdef UNIFY
ret = unifyfs_mount("/unifyfs", myrank, nranks, 0);
if (ret) {
printf("[%d] unifyfs_mount failed (return = %d)\n", myrank, ret);
MPI_Abort(MPI_COMM_WORLD, 1);
}
#endif

if (nranks != 1) {
fprintf(stderr, "Run this program on one process only\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}

fprintf(stderr,
"This program creates an 4 Gbyte file. Don't run it if you don't have that much disk space!\n");

buf = (long long *) malloc(SIZE * sizeof(long long));
if (!buf) {
fprintf(stderr, "not enough memory to allocate buffer\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}

MPI_File_open(MPI_COMM_SELF, filename,
MPI_MODE_CREATE | MPI_MODE_RDWR, MPI_INFO_NULL, &fh);

for (i = 0; i < NTIMES; i++) {
for (j = 0; j < SIZE; j++)
buf[j] = i * SIZE + j;

err = MPI_File_write(fh, buf, SIZE, MPI_DOUBLE, &status);
/* MPI_DOUBLE because not all MPI implementations define
* MPI_LONG_LONG_INT, even though the C compiler supports long long. */
if (err != MPI_SUCCESS) {
fprintf(stderr, "MPI_File_write returned error\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}
}

sleep(2);
MPI_File_get_size(fh, &size);
fprintf(stderr, "file size = %lld bytes\n", size);

MPI_File_seek(fh, 0, MPI_SEEK_SET);

for (j = 0; j < SIZE; j++)
buf[j] = -1;

flag = 0;
for (i = 0; i < NTIMES; i++) {
err = MPI_File_read(fh, buf, SIZE, MPI_DOUBLE, &status);
/* MPI_DOUBLE because not all MPI implementations define
* MPI_LONG_LONG_INT, even though the C compiler supports long long. */
if (err != MPI_SUCCESS) {
fprintf(stderr, "MPI_File_real returned error\n");
MPI_Abort(MPI_COMM_WORLD, 1);
}
for (j = 0; j < SIZE; j++)
if (buf[j] != i * SIZE + j) {
fprintf(stderr, "error: buf %d is %lld, should be %lld \n", j, buf[j],
i * SIZE + j);
flag = 1;
}
}

if (!flag)
fprintf(stderr, "Data read back is correct\n");
MPI_File_close(&fh);

free(buf);
#ifdef UNIFY
unifyfs_unmount();
#endif
MPI_Finalize();
return 0;
}
```

Reproduce the one-process 4GB case from src/mpi/romio/test/large_file.c.in, then inspect the MPI_File_get_size path after the 128 writes. Compare the 2GB and 4GB runs and the unifyfs_intercept_fd() and unifyfs_sync() log errors; done means the 4GB test reports the expected size and reads the data back correctly.

Written by the indexing model from the issue text.