commaai / commaai/research

Training transition model is too resource intensive, uses too much memory. Possible bug

Open
#27 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.1k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

After [training the autoencode](https://github.com/commaai/research/blob/master/DriveSim.md), i try to train the transition model as described by the same document.

using

```
./server.py --time 60 --batch 64
```

and

```
./train_generative_model.py transition --batch 64 --name transition
```

on two different tmux sessions.

Soon (a minute) after running the training command, the process is killed because my memory and swap (16 + 10 GB) are used up, and I'm still on epoch one.

Here is a dump:

```
/train_generative_model.py transition --batch 64 --name transition [0/0]
Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 1060 6GB
major: 6 minor: 1 memoryClockRate (GHz) 1.7085
pciBusID 0000:01:00.0
Total memory: 5.93GiB
Free memory: 5.58GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)
T.shape: (64, 14, 512)
Transition variables:
transition/dreamyrnn_1_W:0
transition/dreamyrnn_1_U:0
transition/dreamyrnn_1_b:0
transition/dreamyrnn_1_V:0
transition/dreamyrnn_1_ext_b:0
Epoch 1/200
Killed
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.