stan-dev / stan-dev/math

Increase flexiblility of MPI task partition patterns

Open
#948 10 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature mpi
Dominant language
C++
Stars
839
Forks
220
Avg merge
2d 4h
Merged PRs (30d)
14

Description

Summary:

Allow more partition patterns in MPI jobs, instead of single equally-divided job allocation pattern.

Description:

map_rect only allows equally-divided pattern for MPI job allocation. That is, in mpi_map_chunks the number of jobs in each node is calculated as total_number_of_jobs/total_number_of_nodes. This is assuming each job is small enough to be efficiently handled by a single node. For many problems this assumption is not met, such examples include user-supplied PDE solution routine. For such an application, we may only have a single job for each chain, but that job requires N nodes to process, where N>>1. Current mpi_map_chunks function will map this single job to a single node, and leave the rest N-1 node idle, instead of delegating the whole N nodes to the task(how the N nodes are used to perform the task depends on application).

A related issue is #940 , where we allow specifying communicator to construct mpi_cluster, so that a group of dedicated nodes can be used in a task-specific communication group.

For example, we may look at an input like the following.

functions {
   real[] user_function_based_on_metis(...);
  }
}
parameters {
real theta;
...
}
....
model {
  ....
  target += sum(map_rect(user_function_based_on_metis, beta, theta, xs, ys));
}

Since there is only one theta, no matte how many nodes we throw in, only the root will be used to perform the task of evaluating user_function_based_on_metis, the heavy-lifting function using theta to solve a problem using MPI partition produced by METIS. What SHOULD happen is ALL the nodes issued are used to evaluate this function.

TL;DR
Current mpi_map_chunks pattern is too rigid, we need fix loosen it.

Current Version:

v2.18.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the mpi_map_chunks allocation logic described in the issue and compare it with the related communicator work in issue #940. Define the supported partition patterns and verify that a task requiring N nodes can receive all N nodes rather than leaving N-1 idle; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.