CliMA / CliMA/ClimaCore.jl

Distributed regridding - store local weights and indices on each process

Open
#1,195 0 comments 0 reactions 0 assignees View on GitHub
distributed
Dominant language
Julia
Stars
117
Forks
19
Avg merge
3d 4h
Merged PRs (30d)
41

Description

**Is your feature request related to a problem? Please describe.**
As of #1192, each process has access to only its local target indices when performing a remapping. This simplifies the logic in `remap!` and isolates setup to `generate_map`. However, each process still stores the entire index vectors, just with 0s at non-local indices.

Here we wish to further expand on this idea by storing only the local segments of the source and target indices, weights, and row indices in the `LinearMap` of each process.

Part of SDI [#188](https://github.com/CliMA/ClimaCoupler.jl/issues/188).

**Describe the solution you'd like**
We need to change the distributed storage of 4 quantities: `source_local_idxs`, `target_local_idxs`, `weights`, `row_indices`.

For each element in these vectors, the root process needs to be aware of the `pid` of the process responsible for that element. We can find this usign the `Topology2D.elem_pid` vector. We can then use this to aggregate the information for each process into a matrix, and perform one broadcast step where we send these matrices each to their respective process.

Specific steps to do this are:
1. When looping over the weights vector, check the pid of `et` (target element) using `elem_pid`, and add `et` and the current weight to the matrix intended for pid `n`.
2. At the end of `generate_map`, distribute these matrices to the corresponding processes.

* Not sure how to select source indices and make sure to send them to the right place - this may require the super-halo. Either way, we need to make sure that the ordering and length of the index and weight vectors allows us to perform the remapping multiplication correctly.

**Describe alternatives you've considered**
MPI's `scatter` function _almost_ does what we want. The issue here is that the values we need to store all use TempestRemap's indexing, so we aren't sure that the partitioning used by `scatter` will correctly put the values on the processes they need to be on.

**Additional context**

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace the distributed path from generate_map into each process's LinearMap, then compare it with remap! and the local-index behavior from #1192. Inspect how Topology2D.elem_pid relates target elements and how source_local_idxs, target_local_idxs, weights, and row_indices are consumed. Done means each process stores only its local values while preserving the ordering and lengths required for remapping multiplication.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.