llnl / llnl/SAMRAI

Refined level fails to get periodic patch ghosts values

Open
#311 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
250
Forks
89
Avg merge
34m
Merged PRs (30d)
1

Description

The following describes what appears to be a bug in SAMRAI.
In a (50,50) cells tagged hierarchy with max number of levels =2 and 2 MPI Ranks and fully periodic domain I end up with 1 refined level in the following configuration:

Image

Level 0:

[Patch: box( Box([0, 25],[49, 49])), id(p0#0),
Patch: box( Box([0, 0],[49, 24])), id(p1#1)]

Level 1:

Patch: box( Box([0, 48],[99, 79])), id(p0#0),
Patch: box( Box([0, 16],[99, 47])), id(p1#1)

dashed lines are the L1 patches.
L0 patches are separated in the middle vertically.

Thus Level 1 has 2 patches, each touching the X left and right periodic boundaries.

However, when filling ghost values on L1 data, left and right borders are obtained sometimes obtained from refinement from L0 instead of periodic overlaps.

The trigger seems to be the smallest_patch_size parameters. With (3,3) we do get ghosts from periodic L1 values, whereas for (5,5) we do not. Our patch Data has (2,2) ghosts nodes.

The issue only appears in parallel as far as I can tell. L1 gets its ghosts correctly from periodic L1 overlap in sequential runs although L1 boxes are the same.

Debugging deep into SAMRAI's code reveals that differences appear already in makeFinerLevel>findRefinementBoxes>bridgeWithNesting where the tag_to_tag connector that is passed has a width that differs from (6,6) to (7,7). d_tag_to_cluster_width also differs. This results in things differing in visible_west_nabrs and in the end neighbor periodic box images are not taken into account when the ghost schedule is created and it does not call caculateOverlap()...

While I understand that changing the smallest patch size may changes some internal calculations, I do not think this should change the fact that L1 patches touching periodic borders get or not values from periodic overlaps while the L1 patch boxes etc. are exactly the same.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at makeFinerLevel>findRefinementBoxes>bridgeWithNesting and inspect how tag_to_tag and d_tag_to_cluster_width are computed for the reported parallel configuration. Reproduce the two smallest_patch_size cases with two MPI ranks, then compare periodic L1 ghost sources and neighbor images. Done means periodic L1 overlaps are used consistently for both (3,3) and (5,5), without changing the patch boxes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.