AmgX not determinstic on large matrices even with determinism_flag=1
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 692
- Forks
- 197
- PR merge metrics
- No merged PRs in 30d
Description
I have a large matrix (approximately 8 million rows) for which AmgX doesn't produce deterministic results. Running the exact same programme twice with the exact same input will produce slight differences in the solution vector. These differences are on the order of 1e-14. Nevertheless, with determinism_flag=1 the solution should be entirely deterministic and give exactly the same results right? So far I've only done detailed testing on a single GPU, but some other tests I did suggested this was also occurring for multi-GPU runs too.
It looks like the non-determinism occurs when constructing the multi-grid hierarchy as when I print the grid stats I can see slight differences between the number of rows and non-zero entries in some of the grid levels between runs.
I'm happy to provide a minimum working example and the matrix I'm using if you want me to. Just let me know.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with a minimum working example and the approximately 8-million-row matrix, then run it twice with determinism_flag=1 while recording the multigrid hierarchy statistics and solution vectors. Compare single-GPU and multi-GPU runs if available; done means the hierarchy and resulting solution are identical between repeated runs, or the remaining source of variation is identified.
Written by the indexing model from the issue text.
Assessment
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100