The Java Tensorflow GPU library has a memory leak.

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_memory_leak

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):YES
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:NO
• TensorFlow installed from (source or binary): from https://oss.sonatype.org/
• TensorFlow version (use command below):TensorFlow 2.4.1 / Java 0.3.1-gpu
• Python version:NO
• Bazel version (if compiling from source):NO
• GCC/Compiler version (if compiling from source):NO
• CUDA/cuDNN version: cuda_11.2.1/libcudnn8_8.1.0.77-1/libcudnn8-dev_8.1.0.77-1
• GPU model and memory: GeForce RTX 3070TI / 7981MiB
• Driver Version: 460.84

Describe the current behavior
CPU version: No memory leak.(tensorflow-core-platform-cpu:0.3.1)
GPU version: memory leak occurred. (tensorflow-core-platform-gpu:0.3.1)

No changes have been made to the code.
In each round of execution, the GPU library did not release the memory correctly, and the memory continued to increase until an exception occurred.

Describe the expected behavior

Code to reproduce the issue
The most basic graph operations.

Other info / logs
2021-06-21 03:37:32.085884: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-21 03:37:32.252155: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-06-21 03:37:32.288428: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-21 03:37:32.289228: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:03:00.0 name: GeForce RTX 3070 Ti computeCapability: 8.6
coreClock: 1.77GHz coreCount: 48 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 566.30GiB/s
2021-06-21 03:37:32.289318: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-21 03:37:32.292972: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-06-21 03:37:32.293014: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-06-21 03:37:32.294163: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-06-21 03:37:32.294452: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-06-21 03:37:32.297755: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-06-21 03:37:32.298788: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-06-21 03:37:32.298964: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-06-21 03:37:32.299075: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-21 03:37:32.299705: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-21 03:37:32.300236: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-21 03:37:32.300275: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-21 03:37:32.913810: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-21 03:37:32.913854: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-06-21 03:37:32.913866: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-06-21 03:37:32.914029: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-21 03:37:32.914613: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-21 03:37:32.915568: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-21 03:37:32.916086: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6731 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070 Ti, pci bus id: 0000:03:00.0, compute capability: 8.6)
2021-06-21 03:37:33.131901: I external/org_tensorflow/tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2799830000 Hz
2021-06-21 03:39:47.360919: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-06-21 03:39:48.122008: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-06-21 03:39:48.124108: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
Round 0 learn:7206..7566 test:7926..8286 millis:2078
Round 1 learn:7207..7567 test:7927..8287 millis:1219
....
Round 28 learn:7234..7594 test:7954..8314 millis:840
Round 29 learn:7235..7595 test:7955..8315 millis:805

Exception in thread "main" java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes (37672M) > maxPhysicalBytes (32176M)
at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:695)
at org.tensorflow.internal.c_api.AbstractTF_Tensor.withDeallocator(AbstractTF_Tensor.java:98)
at org.tensorflow.Session.run(Session.java:694)
at org.tensorflow.Session.access$100(Session.java:72)
at org.tensorflow.Session$Runner.runHelper(Session.java:381)
at org.tensorflow.Session$Runner.run(Session.java:329)

Start with the reported stack trace at Session.run, Session.Runner.run, AbstractTF_Tensor.withDeallocator, and Pointer.deallocator, comparing repeated GPU rounds with the CPU version. The issue does not include a reproducer or expected behavior, so first establish the minimal graph-operation case. Done means repeated execution no longer causes physical memory usage to grow until OutOfMemoryError.