NVIDIA / NVIDIA/AMGX

bug with aggressive_levels=1

Open
#222 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
692
Forks
197
PR merge metrics
No merged PRs in 30d

Description

Dear developers, I want to report a bug that I found on a100 once aggressive_levels=1.
Here is the message
AMGX version 2.4.0
Built on Nov 22 2022, 14:07:12
Compiled with CUDA Runtime 11.5, using CUDA driver 11.6
The AMGX_initialize_plugins API call is deprecated and can be safely removed.
Cannot read file as JSON object, trying as AMGX config
Converting config string to current config version
Parsing configuration string: selector=AGGRESSIVE_PMIS ;
Thrust failure: trivial_device_copy D->H failed: cudaErrorIllegalAddress: an illegal memory access was encountered
File and line number are not available for this exception.
Caught amgx exception: Error, setup must be called before calling solve
at: /home/nvarini/arm-gnu-stack/AMGX/src/solvers/solver.cu:599
Stack trace:

I am using 1) gcc/11.2.0-cuda 2) openmpi/4.1.2-cuda 3) cuda/11.5.1 4) hdf5/1.13.0-mpi
At the link https://drive.google.com/file/d/1U96S96UIXSA6dAfhuqLGVlGTM2zdH1_p/view?usp=share_link
there is a minimalistic problem that reproduce the error. If aggressive_levels=0 the application run correctly.
This bug doesn't appear on V100.

Regards

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked minimal reproduction and compare its behavior with aggressive_levels=0 and aggressive_levels=1 on A100 and V100 hardware. Review the AMGX setup and solver path involved in the reported illegal memory access; done means the reproduction runs successfully with aggressive_levels=1 on the affected configuration without the CUDA error.

Written by the indexing model from the issue text.

Assessment

Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.