Division by zero error at random places if GPU is used

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

binary

TensorFlow version

2.18

Custom code

Yes

OS platform and distribution

Linux, RedHatEnterprise 8.6

Mobile device

No response

Python version

N/A

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

12.3

GPU model and memory

Quadro RTX 6000

Current behavior?

(I do not use Python and found no nightly build for C API)
I have a simple program that builds graphs through the c_api and executes sessions. It works perfectly as long as it is run only on CPU. If GPU is involved then at random places the program generates a Div0 error. The very same program run twice within one minute on the same hardware, etc. behave differently.
The error is somewhere deep in TF/Cuda because in the error dump I see 15 program jumps outside my code (what I can still debug).
I tried it on the different partition of a HPC, but both behave the same way. Card is Quadro RTX 6000, driver 565.57.01 Cuda 12.7 as seen in nvidia-smi, but 12.3 is available as libraries.
I tried with many different settings, etc., but cannot identify any rootcause. I am not even sure if the bug is in the TF binary or in one of the Cuda libraries (or elsewhere).

Standalone code to reproduce the issue

I use a Pascal program, called examples, available here: https://github.com/zsoltszakaly/tensorflowforpascal.
It is compiled on the HPC with fpc -MObjFPC -Sh -Fl../tensorflow/lib examples.pas.
The tensorflow/lib directory has
Jan  1  2000  libtensorflow_framework.so -> libtensorflow_framework.so.2
Dec 10 11:04  libtensorflow_framework.so.2 -> libtensorflow_framework.so.2.18.0
Jan  1  2000  libtensorflow_framework.so.2.18.0
Jan  1  2000  libtensorflow.so -> libtensorflow.so.2
Dec 10 11:00  libtensorflow.so.2 -> libtensorflow.so.2.18.0
Jan  1  2000  libtensorflow.so.2.18.0

Relevant log output

It generates every time a session is run, correctly this:
I0000 00:00:1733834127.549506 1676833 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38484 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:41:00.0, compute capability: 8.0

But in the last one (again, it is random when!):
An unhandled exception occurred at $00007F223F261EA1:
EDivByZero: Division by zero
  $00007F223F261EA1
  $00007F223F1E2E72
  $00007F223F1E1D5D
  $00007F223F1E4183
  $00007F223F1E4AD2
  $00007F223F1E6DAE
  $00007F223F1D842B
  $00007F223F1D4241
  $00007F223EA97604
  $00007F223EA9608F
  $00007F223EA921E6
  $00007F223EA9067A
  $00007F223EA90231
  $00007F222E718923
  $00007F222E72561B
  $00000000004626AA
  $0000000000462931