tcmalloc: large alloc on Colab and Tensorflow killed on local machine due to over consumption of RAM

System information

• What is the top-level directory of the model you are using: /home
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): 1.9.0
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: 10.1.243
• GPU model and memory: NVIDIA Quadro RTX 5000; and 16 GB RAM
• Exact command to reproduce:
  I ran the following code in an ipython notebook in both my local machine (local GPU) and Google Colab :

!git clone https://github.com/charlesq34/pointnet.git
cd pointnet/sem_seg/
!sh download_data.sh
!python train.py --log_dir log6 --test_area 6

Describe the problem

The tensorflow API always tries to consume the maximum RAM even when I have a GPU and the kernel gets killed while training my deep learning algorithm. I referred online on multiple sources (1, 2, 3, 4, 5, 6) and tried the following things :

1. Reduce the batch size
2. Change the optimizer from adam to momentum

However, none of these suggestions helped to solve the problem.

Source code / logs

The error log is very long and hence I am attaching it in a separate text file here :
ERROR_LOG.txt