running on A10/A40/A100 not working

We are not able to get the latest AF build running on our high end GPUs, meaning we are running into memory issues on the low end machines.

I am using newer cuda (11.8, 12.x) and cudnn (8.6, 8.8).
Moreover, I took special care about finding a compatible version of
tensorflow and jax, and tested also several combinations. The current
situation is, it does not recognized cuda, the log file output is below:

tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed
call to cuInit: CUDA_ERROR_SYSTEM_NOT_READY: system not yet initialized
I0411 22:49:28.356107 23069551249216 templates.py:857] Using precomputed
obsolete pdbs
/nfs/scistore14/rcsb/alphafold.databases2/pdb_mmcif/obsolete.dat.
2023-04-11 22:49:28.522194: E
external/xla/xla/stream_executor/cuda/cuda_driver.cc:268] failed call to
cuInit: CUDA_ERROR_SYSTEM_NOT_READY: system not yet initialized
I0411 22:49:28.532533 23069551249216 xla_bridge.py:440] Unable to
initialize backend 'cuda': FAILED_PRECONDITION: No visible GPU devices.

and the job is using the cpu and not the gpu.

I tend to believe that the software stack on top of cuda 11.8 and higher
is not so well tested, and that something in cuda/cudnn/tensorflow/xla
is faulty. It could be also that the checks in alphafold for these are
broken. So it is not clear to me whether it is a bug in alphafold or in
tensorflow.

Thank you for your help