NVIDIA / NVIDIA/AMGX

AmgX not determinstic on large matrices even with determinism_flag=1

Open
#121 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug tracking internally
Dominant language
Cuda
Stars
692
Forks
197
PR merge metrics
No merged PRs in 30d

Description

I have a large matrix (approximately 8 million rows) for which AmgX doesn't produce deterministic results. Running the exact same programme twice with the exact same input will produce slight differences in the solution vector. These differences are on the order of 1e-14. Nevertheless, with determinism_flag=1 the solution should be entirely deterministic and give exactly the same results right? So far I've only done detailed testing on a single GPU, but some other tests I did suggested this was also occurring for multi-GPU runs too.

It looks like the non-determinism occurs when constructing the multi-grid hierarchy as when I print the grid stats I can see slight differences between the number of rows and non-zero entries in some of the grid levels between runs.

I'm happy to provide a minimum working example and the matrix I'm using if you want me to. Just let me know.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with a minimum working example and the approximately 8-million-row matrix, then run it twice with determinism_flag=1 while recording the multigrid hierarchy statistics and solution vectors. Compare single-GPU and multi-GPU runs if available; done means the hierarchy and resulting solution are identical between repeated runs, or the remaining source of variation is identified.

Written by the indexing model from the issue text.

Assessment

Domain
hpc
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.