NVIDIA / NVIDIA/AMGX

Classical AMG struggling to converge in distributed mode for singular system

Open
#100 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

usage question
Dominant language
Cuda
Stars
692
Forks
197
PR merge metrics
No merged PRs in 30d

Description

First of all, many thanks for building this library. It is very useful and your hard work and effort is very much appreciated.

I have a Pressure Poisson Equation with pure Neumann boundary conditions (singular system) for which a solution exists up to an arbitrary constant. The matrix file (.mtx) and source file (with configuration settings) are attached to reproduce the below tests.

I realise that this probably isn't an application where AmgX would be expected to perform well. Nevertheless, for a single GPU, AmgX converges to a solution with no problems.

Single GPU

         Number of Levels: 2
            LVL         ROWS               NNZ    SPRSTY       Mem (GB)
         --------------------------------------------------------------
           0(D)         2601             56637   0.00837       0.000698
           1(D)          483             13183    0.0565       0.000309
         --------------------------------------------------------------
         Grid Complexity: 1.1857
         Operator Complexity: 1.23276
         Total Memory Usage: 0.00100739 GB
         --------------------------------------------------------------
         Total Iterations: 14
         Avg Convergence Rate: 		         0.1242
         Final Residual: 		   1.012780e-05
         Total Reduction in Residual: 	   2.077555e-13
         Maximum Memory Usage: 		          0.916 GB
         --------------------------------------------------------------
Total Time: 0.026293
    setup: 0.015236 s
    solve: 0.011057 s
    solve(per iteration): 0.000789785 s

However, when testing the same problem with the same configuration settings on two GPUs, AmgX does not converge to a solution.

Two GPUs

         Number of Levels: 2
            LVL         ROWS               NNZ    SPRSTY       Mem (GB)
         --------------------------------------------------------------
           0(D)         2601             56637   0.00837       0.000756
           1(D)          486             13222     0.056       0.000394
         --------------------------------------------------------------
         Grid Complexity: 1.18685
         Operator Complexity: 1.23345
         Total Memory Usage: 0.00115038 GB
         --------------------------------------------------------------
         Total Iterations: 100
         Avg Convergence Rate: 		         0.8936
         Final Residual: 		   6.361270e+02
         Total Reduction in Residual: 	   1.304912e-05
         Maximum Memory Usage: 		          1.345 GB
         --------------------------------------------------------------
Total Time: 0.459247
    setup: 0.0367741 s
    solve: 0.422472 s
    solve(per iteration): 0.00422472 s

After playing around with the configuration settings I can get a converged solution by increasing the iterations in the AMG preconditioner step. However, the solve time is approximately 100x the single GPU case.

Two GPUs (precon:max_iters=100)

         Total Iterations: 4
         Avg Convergence Rate: 		         0.0001
         Final Residual: 		   1.430485e-08
         Total Reduction in Residual: 	   2.934410e-16
         Maximum Memory Usage: 		          1.336 GB
         --------------------------------------------------------------
Total Time: 1.19626
    setup: 0.0366399 s
    solve: 1.15962 s
    solve(per iteration): 0.289904 s

To get around this I can obviously pin the solution at a particular point to make the matrix non-singular (which I have tested). However, this isn't always an ideal solution.

From reading the documentation I can see that the distributed version of the Classical AMG is slightly weaker due to coupling issues. Also, like I said before, I wouldn't have expected AmgX to perform particularly well for this type of problem. Nevertheless, I thought the difference in performance between the single-GPU and distributed versions was interesting and was wondering if there are any particular settings which might help with the coupling or overall performance of the distributed version?
main.cpp.txt
System.mtx.txt

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the attached main.cpp.txt with System.mtx.txt and compare the single-GPU and two-GPU Classical AMG results using the reported configuration. Read the distributed AMG settings and convergence output first; done means the distributed case's convergence and performance difference are explained and a specific helpful setting or change is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.