NVIDIA / NVIDIA/AMGX

How to use AMGX_matrix_upload_distributed with external MPI initialization

Open
#144 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
692
Forks
197
PR merge metrics
No merged PRs in 30d

Description

Good day. I am trying to apply the distributed matrix loading approach from the example AMGX / examples / amgx_mpi_capi_cla.c in my code. The peculiarity of my code is that I have MPI initialization in external code. On each processor, I work with my own piece of the grid, and I use AMGX only to calculate the SLAE. Can you please tell me how in this case it is worth modifying the example to make it work?
I tried it in different ways. In my current configuration, I have it like this:
MPI_Comm_dup (MPI_COMM_WORLD, & amgx_mpi_comm);
/ * MPI init (with CUDA GPUs) * /
// MPI
MPI_Comm_size (amgx_mpi_comm, & nranks);
MPI_Comm_rank (amgx_mpi_comm, & rank);
// CUDA GPUs
CUDA_SAFE_CALL (cudaGetDeviceCount (& gpu_count));
lrank = rank% gpu_count;
CUDA_SAFE_CALL (cudaSetDevice (lrank));
printf ("Process% d selecting device% d \ n", rank, lrank);

     // app must know how to provide a mapping
     AMGX_resources_create (& rsrc, config, & amgx_mpi_comm, 1, & lrank);

      if (partition_vector == NULL)
                  {
                      // If no partition vector is given, we assume a partitioning with contiguous blocks (see example above). It is sufficient (and faster/more scalable)
            // to calculate the partition offsets and pass those into the API call instead of creating a full partition vector.
            int64_t* partition_offsets = (int64_t*)malloc((nranks + 1) * sizeof(int64_t));
            // gather the number of rows on each rank, and perform an exclusive scan to get the offsets.
            int64_t n64 = n;
            partition_offsets[0] = 0; // rows of rank 0 always start at index 0
            MPI_Allgather(&n64, 1, MPI_INT64_T, &partition_offsets[1], 1, MPI_INT64_T, amgx_mpi_comm);
            for (int i = 2; i < nranks + 1; ++i) {
                partition_offsets[i] += partition_offsets[i - 1];
            }
            nglobal = partition_offsets[nranks]; // last element always has global number of rows

            AMGX_distribution_handle dist;
            AMGX_distribution_create(&dist, config);
            AMGX_distribution_set_partition_data(dist, AMGX_DIST_PARTITION_OFFSETS, partition_offsets);
            AMGX_matrix_upload_distributed(A, nglobal, n, nnz, block_dimx, block_dimy, row_ptrs, col_indices, values, diag, dist);
            AMGX_distribution_destroy(dist);
            free(partition_offsets);
        }

I run execution on 2 processors:

mpiexec -n 2 <application_name>.exe
The log is like this:
Process 0 selecting device 0
...
AMGX version 2.1.0.131-opensource
Built on Aug 17 2020, 11:01:51
Compiled with CUDA Runtime 11.0, using CUDA driver 11.3
Using Normal MPI (Hostbuffer) communicator...

job aborted:
rank: node: exit code[: error message]
0: LAPTOP-L6OI9HRJ: -1073741819: process 0 exited without calling finalize
1: LAPTOP-L6OI9HRJ: -1073741819: process 1 exited without calling finalize

Maybe someone came across such a situation. Any help would be welcome.

ps I read https://github.com/NVIDIA/AMGX/issues/81, I have provided the specifics of matrix formation for this case.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with AMGX / examples / amgx_mpi_capi_cla.c and compare its MPI setup with the external MPI initialization shown. Reproduce the two-process run with mpiexec -n 2 and inspect the AMGX_resources_create and AMGX_matrix_upload_distributed calls. Done means the externally initialized MPI communicator loads the distributed matrix and completes without the reported abort.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.