llnl / llnl/UnifyFS

PnetCDF mcoll_perf detects incorrect data

Open
#757 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pnetcdf
Dominant language
C
Stars
122
Forks
34
PR merge metrics
No merged PRs in 30d

Description

The ``test/nonblocking/mcoll_perf.c`` test detects incorrect data when comparing two files that were written two different ways which should have identical content.
```
cd test/nonblocking
srun -n2 ./mcoll_perf /unifyfs/testfile.nc

P0: diff at line 282 variable[2] var1_2: NC_INT buf1 != buf2 at position 32762
```
After tracing ``pwrite`` and ``pread`` calls under a debugger, the problem is that both ranks write to the same byte offsets without any synchronization in between. In this case, rank 1 writes a fill value and rank 0 later writes actual data. It's a race as to which value actually ends up in the file.

The fill call is here:

https://github.com/Parallel-NetCDF/PnetCDF/blob/bb59553ca3542bc09ead12c6ce4e65b913ef51fa/test/nonblocking/mcoll_perf.c#L521

When filling the variable 2, rank 1 writes to (offset=648, length=8) and (offset=680, length=8).

And the write call is here:

https://github.com/Parallel-NetCDF/PnetCDF/blob/bb59553ca3542bc09ead12c6ce4e65b913ef51fa/test/nonblocking/mcoll_perf.c#L526

In that write, rank 0 writes to (offset=640, length=16) and (offset=672, length=16), which overlaps with the region that rank 1 wrote to during the fill operation.

The test case can be fixed by adding a call to ``ncmpi_sync(ncid);``:
```
for (i=2; i

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test/nonblocking/mcoll_perf.c at the fill loop around line 521 and the nonblocking collective writes around line 526. Run `cd test/nonblocking` and `srun -n2 ./mcoll_perf /unifyfs/testfile.nc` to reproduce the incorrect comparison, then verify that synchronization prevents overlapping fill and write operations from racing and the test completes without the reported data mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.