Can't use UMAP after having been using it for a while

Nobody has claimed this yet.

I have been using bertopic for training topic models during the past months, and it worked fine.
However, since last week, I have been running into issues with making it run on the clusters from my university.
I kept getting errors as follows, and I couldn't find any similar issues here. It seems like a cuBLAS version is incompatible but I am not sure how I can fix it. This error is only thrown after running the script for around 40 minutes. That's why I also reduced the n_neighbors parameter in the UMAP function, which was running fine before.

Any suggestions will be appreciated!

Traceback (most recent call last):
File "/var/scratch/xli3/workspace/dstm/dstm/bert-topic.py", line 104, in
topics, probs = topic_model.fit_transform(corpus)
File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/bertopic/_bertopic.py", line 386, in fit_transform
umap_embeddings = self._reduce_dimensionality(embeddings, y)
File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/bertopic/_bertopic.py", line 3183, in _reduce_dimensionality
self.umap_model.fit(embeddings, y=y)
File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper
ret = func(*args, **kwargs)
File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 393, in dispatch
return self.dispatch_func(func_name, gpu_func, *args, **kwargs)
File "/var/scratch/xli3/miniconda3/envs/dstm2/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 190, in wrapper
return func(*args, **kwargs)
File "base.pyx", line 674, in cuml.internals.base.UniversalBase.dispatch_func
File "umap.pyx", line 603, in cuml.manifold.umap.UMAP.fit
RuntimeError: cuBLAS error encountered at: file=/__w/cuml/cuml/python/_skbuild/linux-x86_64-3.10/cmake-build/_deps/raft-src/cpp/include/raft/core/resource/cublas_handle.hpp line=75: call='cublasSetStream(ret, get_cuda_stream(res))', Reason=1:CUBLAS_STATUS_NOT_INITIALIZED

Start with the failing call at line 104 of bert-topic.py and trace BERTopic.fit_transform into _reduce_dimensionality and the cuML UMAP.fit traceback shown in the report. Reproduce the failure after the reported runtime and compare the CUDA/cuBLAS, cuML, and UMAP environment details; done means identifying a confirmed cause and a reproducible fix or documented compatibility requirement.

Written by the indexing model from the issue text.