Classical AMG struggling to converge in distributed mode for singular system
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 692
- Forks
- 197
- PR merge metrics
- No merged PRs in 30d
Description
First of all, many thanks for building this library. It is very useful and your hard work and effort is very much appreciated.
I have a Pressure Poisson Equation with pure Neumann boundary conditions (singular system) for which a solution exists up to an arbitrary constant. The matrix file (.mtx) and source file (with configuration settings) are attached to reproduce the below tests.
I realise that this probably isn't an application where AmgX would be expected to perform well. Nevertheless, for a single GPU, AmgX converges to a solution with no problems.
Single GPU
Number of Levels: 2
LVL ROWS NNZ SPRSTY Mem (GB)
--------------------------------------------------------------
0(D) 2601 56637 0.00837 0.000698
1(D) 483 13183 0.0565 0.000309
--------------------------------------------------------------
Grid Complexity: 1.1857
Operator Complexity: 1.23276
Total Memory Usage: 0.00100739 GB
--------------------------------------------------------------
Total Iterations: 14
Avg Convergence Rate: 0.1242
Final Residual: 1.012780e-05
Total Reduction in Residual: 2.077555e-13
Maximum Memory Usage: 0.916 GB
--------------------------------------------------------------
Total Time: 0.026293
setup: 0.015236 s
solve: 0.011057 s
solve(per iteration): 0.000789785 s
However, when testing the same problem with the same configuration settings on two GPUs, AmgX does not converge to a solution.
Two GPUs
Number of Levels: 2
LVL ROWS NNZ SPRSTY Mem (GB)
--------------------------------------------------------------
0(D) 2601 56637 0.00837 0.000756
1(D) 486 13222 0.056 0.000394
--------------------------------------------------------------
Grid Complexity: 1.18685
Operator Complexity: 1.23345
Total Memory Usage: 0.00115038 GB
--------------------------------------------------------------
Total Iterations: 100
Avg Convergence Rate: 0.8936
Final Residual: 6.361270e+02
Total Reduction in Residual: 1.304912e-05
Maximum Memory Usage: 1.345 GB
--------------------------------------------------------------
Total Time: 0.459247
setup: 0.0367741 s
solve: 0.422472 s
solve(per iteration): 0.00422472 s
After playing around with the configuration settings I can get a converged solution by increasing the iterations in the AMG preconditioner step. However, the solve time is approximately 100x the single GPU case.
Two GPUs (precon:max_iters=100)
Total Iterations: 4
Avg Convergence Rate: 0.0001
Final Residual: 1.430485e-08
Total Reduction in Residual: 2.934410e-16
Maximum Memory Usage: 1.336 GB
--------------------------------------------------------------
Total Time: 1.19626
setup: 0.0366399 s
solve: 1.15962 s
solve(per iteration): 0.289904 s
To get around this I can obviously pin the solution at a particular point to make the matrix non-singular (which I have tested). However, this isn't always an ideal solution.
From reading the documentation I can see that the distributed version of the Classical AMG is slightly weaker due to coupling issues. Also, like I said before, I wouldn't have expected AmgX to perform particularly well for this type of problem. Nevertheless, I thought the difference in performance between the single-GPU and distributed versions was interesting and was wondering if there are any particular settings which might help with the coupling or overall performance of the distributed version?
main.cpp.txt
System.mtx.txt
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the attached main.cpp.txt with System.mtx.txt and compare the single-GPU and two-GPU Classical AMG results using the reported configuration. Read the distributed AMG settings and convergence output first; done means the distributed case's convergence and performance difference are explained and a specific helpful setting or change is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100