[QST] how have a planned but not CUDA/CAGRA aware exit() while CAGRA is in a library?

My CAGRA stuff is hidden in a library that has a generic data passing C interface (but both sides are C++).

Im trying to have something like this at the top of a .cu file:

    struct cagra_context_s {
    raft::device_resources dev_resources;
    cuvs::neighbors::cagra::index_params index_params;
    cuvs::neighbors::cagra::search_params search_params;
    std::optional<raft::device_matrix<float, int64_t>> dataset_d;
    std::optional<cuvs::neighbors::cagra::index<float, uint32_t>> index;

    void allocate_dataset(int64_t num_vectors, int64_t vec_size) {
    if (!dataset_d) {
    dataset_d.emplace(raft::make_device_matrix<float, int64_t>(dev_resources, num_vectors, vec_size));
    }
    }

    void set_index(cuvs::neighbors::cagra::index<float, uint32_t>&& idx) {
    index.emplace(std::move(idx)); // moves into the optional
    }
    };
    // the key is the mem_id from CUVS.cpp
    std::unordered_map<unsigned long long, cagra_context_s> cagra_context;

but I keep on getting

CUDA Error detected. cudaErrorCudartUnloading driver shutting down app: /home/pmilosla/projects/cuda/cuvs_install/include/rmm/mr/device/cuda_memory_resource.hpp:80: virtual void rmm::mr::cuda_memory_resource::do_deallocate(void*, std::size_t, rmm::cuda_stream_view): Assertion status__ == cudaSuccess’ failed.
Aborted (core dumped)`

during a planned but non CUDA aware exit.

How do I make option 2 work without a core? Any .cu and library code change is allowed

• controlled cuda aware exit (i.e. its decided that it happens in the cuda flow and the user knows enough to call a cagra cleanup everything function) (this works fine)
• controlled but non cuda aware exit (i.e. the data manager decided to exit what they were doing and forgot to issue “clean up cuda”) (will get core)
• an error (not anticipated by any user) (will get core)

More details here: https://forums.developer.nvidia.com/t/cuda-run-time-library-unload/51956/8

Start with the CUVS.cpp entry point and the .cu code defining cagra_context_s, then reproduce the planned non-CUDA-aware exit described in the issue. Trace the lifetimes of cagra_context, its optionals, and device resources during shutdown. Done means the non-CUDA-aware exit completes without the cudaErrorCudartUnloading assertion or a core dump.