FEniCS / FEniCS/dolfinx

Mixed-topology meshes redistribute cells with one NBX consensus round per cell type

Open
#4,402 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C++
Stars
1.2k
Forks
261
Avg merge
1d 19h
Merged PRs (30d)
71

Description

`mesh::impl::partition_cells` (in `cpp/dolfinx/mesh/utils.h`) redistributes cells by calling `graph::build::distribute` once per cell type:

```cpp
for (std::int32_t i = 0; i < num_cell_types; ++i)
{
...
std::tie(cells1[i], src_ranks, original_idx1[i], ghost_owners[i])
= graph::build::distribute(comm, cells[i], {num_cells, num_cell_nodes}, dest_i);
}
```

Each call to `graph::build::distribute` independently discovers its neighbourhood communicator using the NBX consensus algorithm (`MPI::compute_graph_edges_nbx`, see `cpp/dolfinx/graph/partition.cpp`). This is scalable (no arrays the size of the communicator are built, and the communication pattern stays sparse), but it is not free: each call costs at least one non-blocking consensus round.

For the common case of a single-cell-type mesh this is a non-issue (one cell type, one call, one round). For a genuinely mixed-topology mesh (e.g. tetrahedra + hexahedra), each cell type currently pays for its own independent consensus round, even though the destination rank sets for different cell types typically overlap substantially, since they come from the same overall partitioning decision over the same physical domain.

Possible directions:
- Discover the neighbourhood once (e.g. from the union of destination ranks across all cell types, or from one cell type as an over-approximation of the others) and reuse it for the remaining cell types' exchanges.
- Restructure so a single interleaved payload spanning all cell types is redistributed in one `distribute` call.

This only matters for mixed-topology meshes, and only at large enough rank counts that the NBX consensus-round cost is non-negligible relative to the actual data exchange. For the common single-cell-type case there is nothing to fix.

Contributor guide

Open the contributing guide

Research direction

Start in cpp/dolfinx/mesh/utils.h at mesh::impl::partition_cells and trace graph::build::distribute into cpp/dolfinx/graph/partition.cpp, especially MPI::compute_graph_edges_nbx. Compare the per-cell-type exchanges with the proposed shared-neighbourhood or interleaved-payload approaches. Done means mixed-topology redistribution preserves existing results while avoiding an independent NBX consensus round for every cell type.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.