Refined level fails to get periodic patch ghosts values
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 250
- Forks
- 89
- Avg merge
- 34m
- Merged PRs (30d)
- 1
Description
The following describes what appears to be a bug in SAMRAI.
In a (50,50) cells tagged hierarchy with max number of levels =2 and 2 MPI Ranks and fully periodic domain I end up with 1 refined level in the following configuration:
Level 0:
[Patch: box( Box([0, 25],[49, 49])), id(p0#0),
Patch: box( Box([0, 0],[49, 24])), id(p1#1)]
Level 1:
Patch: box( Box([0, 48],[99, 79])), id(p0#0),
Patch: box( Box([0, 16],[99, 47])), id(p1#1)
dashed lines are the L1 patches.
L0 patches are separated in the middle vertically.
Thus Level 1 has 2 patches, each touching the X left and right periodic boundaries.
However, when filling ghost values on L1 data, left and right borders are obtained sometimes obtained from refinement from L0 instead of periodic overlaps.
The trigger seems to be the smallest_patch_size parameters. With (3,3) we do get ghosts from periodic L1 values, whereas for (5,5) we do not. Our patch Data has (2,2) ghosts nodes.
The issue only appears in parallel as far as I can tell. L1 gets its ghosts correctly from periodic L1 overlap in sequential runs although L1 boxes are the same.
Debugging deep into SAMRAI's code reveals that differences appear already in makeFinerLevel>findRefinementBoxes>bridgeWithNesting where the tag_to_tag connector that is passed has a width that differs from (6,6) to (7,7). d_tag_to_cluster_width also differs. This results in things differing in visible_west_nabrs and in the end neighbor periodic box images are not taken into account when the ghost schedule is created and it does not call caculateOverlap()...
While I understand that changing the smallest patch size may changes some internal calculations, I do not think this should change the fact that L1 patches touching periodic borders get or not values from periodic overlaps while the L1 patch boxes etc. are exactly the same.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at makeFinerLevel>findRefinementBoxes>bridgeWithNesting and inspect how tag_to_tag and d_tag_to_cluster_width are computed for the reported parallel configuration. Reproduce the two smallest_patch_size cases with two MPI ranks, then compare periodic L1 ghost sources and neighbor images. Done means periodic L1 overlaps are used consistently for both (3,3) and (5,5), without changing the patch boxes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100