NVIDIA / NVIDIA/AMGX

OpenFOAM2022+AmgX+PETSc-after determining tmp storage requirements for exclusive_scan: cudaErrorInvalidDevice: invalid device ordinal

Open
#202 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
692
Forks
197
PR merge metrics
No merged PRs in 30d

Description

After initial hiccups with building components, I am now facing problems with running icoFoam (OpenFOAM)
with execution dumping core with the following Error message.I am running this on Laptop having GEForce RTX 3060 - GPU memory is 6GB - Appreciate your help

Number of GPU devices :: 1
AMGX version 2.2.0.132-opensource
Built on Aug 4 2022, 23:33:25
Compiled with CUDA Runtime 11.7, using CUDA driver 11.7
Cannot read file as JSON object, trying as AMGX config
Converting config string to current config version
Parsing configuration string: exception_handling=1 ;
Initializing AmgX-p
Initializing AmgX Linear Solver p
terminate called after throwing an instance of 'thrust::system::system_error'
what(): after determining tmp storage requirements for exclusive_scan: cudaErrorInvalidDevice: invalid device ordinal
Caught signal 6 - SIGABRT (abort)
/usr/lib/openfoam/amgx/build/libamgxsh.so : ()+0xf8864c
/lib/x86_64-linux-gnu/libpthread.so.0 : ()+0x14420
/lib/x86_64-linux-gnu/libc.so.6 : gsignal()+0xcb
/lib/x86_64-linux-gnu/libc.so.6 : abort()+0x12b
/lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0x9e911
/lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa38c
/lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa3f7
/lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa6a9
/home/sambath/OpenFOAM/sambath-v2112/platforms/linux64GccDPInt32Opt/lib/libfoam2csr.so : int* thrust::cuda_cub::detail::exclusive_scan_n_impl<thrust::cuda_cub::par_t, int*, long, int*, int, thrust::plus >(thrust::cuda_cub::execution_policythrust::cuda_cub::par_t&, int*, long, int*, int, thrust::plus)+0x32a
/home/sambath/OpenFOAM/sambath-v2112/platforms/linux64GccDPInt32Opt/lib/libfoam2csr.so : AmgXCSRMatrix::setValuesLDU(int, int, int, int, int, int const*, int const*, int, int const*, int const*, double const*, double const*, double const*, double const*)+0x9c1
/home/sambath/OpenFOAM/sambath-v2112/platforms/linux64GccDPInt32Opt/lib/libpetscFoam.so : Foam::amgxSolver::offloadMatrixArrays(AmgXCSRMatrix&, int&, int&, int&) const+0x761
/home/sambath/OpenFOAM/sambath-v2112/platforms/linux64GccDPInt32Opt/lib/libpetscFoam.so : Foam::amgxSolver::scalarSolve(Foam::Field&, Foam::Field const&, unsigned char) const+0x687
/home/sambath/OpenFOAM/sambath-v2112/platforms/linux64GccDPInt32Opt/lib/libpetscFoam.so : Foam::amgxSolver::solve(Foam::Field&, Foam::Field const&, unsigned char) const+0x48
/usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so : Foam::fvMatrix::solveSegregated(Foam::dictionary const&)+0x625
/usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so : Foam::fvMatrix::solveSegregatedOrCoupled(Foam::dictionary const&)+0x498
/usr/lib/openfoam/openfoam2112/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so : Foam::fvMesh::solve(Foam::fvMatrix&, Foam::dictionary const&) const+0x28
icoFoam : ()+0x289e9
/lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main()+0xf3
icoFoam : ()+0x2a28e
Aborted (core dumped)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing icoFoam with the reported OpenFOAM, AmgX, PETSc, CUDA, and RTX 3060 setup. Trace the failure from AmgXCSRMatrix::setValuesLDU and Foam::amgxSolver::offloadMatrixArrays, then determine why the CUDA device ordinal is invalid; done means identifying a confirmed cause and a reproducible resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.