Tensorflow hanging

Hi! When I run `python3 -m language.search_agents.muzero.muzero_main --run_mode learner --optimizer adam`, the process seems to stop doing work (it's been four hours):

```
…
I1128 23:48:06.708257 139860039362368 learner.py:404] recurrent_core/cell_0/state_norm/gamma:0: (512,) |
I1128 23:48:06.708284 139860039362368 learner.py:404] recurrent_core/cell_0/state_norm/beta:0: (512,) |
I1128 23:48:06.708312 139860039362368 learner.py:404] dense/kernel:0: (1024, 32) |
I1128 23:48:06.708340 139860039362368 learner.py:404] dense/bias:0: (32,) |
I1128 23:48:06.708367 139860039362368 learner.py:404] layer_normalization/gamma:0: (32,) |
I1128 23:48:06.708394 139860039362368 learner.py:404] layer_normalization/beta:0: (32,) |
I1128 23:48:06.708422 139860039362368 learner.py:404] output/kernel:0: (32, 29566) |
I1128 23:48:06.708449 139860039362368 learner.py:404] output/bias:0: (29566,) |
I1128 23:48:06.708478 139860039362368 learner.py:404] dense_2/kernel:0: (512, 32) |
I1128 23:48:06.708505 139860039362368 learner.py:404] dense_2/bias:0: (32,) |
I1128 23:48:06.708531 139860039362368 learner.py:404] layer_normalization_2/gamma:0: (32,) |
I1128 23:48:06.708558 139860039362368 learner.py:404] layer_normalization_2/beta:0: (32,) |
I1128 23:48:06.708586 139860039362368 learner.py:404] output/kernel:0: (32, 8) |
I1128 23:48:06.708614 139860039362368 learner.py:404] output/bias:0: (8,) |
I1128 23:48:06.708642 139860039362368 learner.py:404] final/kernel:0: (800, 1024) |
I1128 23:48:06.708669 139860039362368 learner.py:404] final/bias:0: (1024,) |
I1128 23:48:06.708698 139860039362368 learner.py:404] dense_1/kernel:0: (1024, 32) |
I1128 23:48:06.708725 139860039362368 learner.py:404] dense_1/bias:0: (32,) |
I1128 23:48:06.708752 139860039362368 learner.py:404] layer_normalization_1/gamma:0: (32,) |
I1128 23:48:06.708779 139860039362368 learner.py:404] layer_normalization_1/beta:0: (32,) |
I1128 23:48:06.708807 139860039362368 learner.py:404] output/kernel:0: (32, 8) |
I1128 23:48:06.708835 139860039362368 learner.py:404] output/bias:0: (8,) |
I1128 23:48:06.708863 139860039362368 learner.py:404] rnn/gru_cell/kernel:0: (29567, 96) |
I1128 23:48:06.708892 139860039362368 learner.py:404] rnn/gru_cell/recurrent_kernel:0: (32, 96) |
I1128 23:48:06.708920 139860039362368 learner.py:404] rnn/gru_cell/bias:0: (2, 96) |
I1128 23:48:06.708944 139860039362368 learner.py:405] --------------------------
I1128 23:48:06.710096 139860039362368 learner.py:461] Restoring checkpoint: /tmp/agent/learner/ckpt-1
2021-11-28 23:48:09.319611: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
WARNING:tensorflow:From /home/kevin/pdev/lmrl/language/muzero/learner.py:668: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W1128 23:48:09.942079 139860039362368 deprecation.py:333] From /home/kevin/pdev/lmrl/language/muzero/learner.py:668: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
WARNING:tensorflow:From /home/kevin/pdev/lmrl/language/language/search_agents/env/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py:1119: batch_scatter_update (from tensorflow.python.ops.state_ops) is deprecated and will be removed after 2018-11-29.
Instructions for updating:
Use the batch_scatter_update method of Variable instead.
W1128 23:48:10.096201 139860039362368 deprecation.py:333] From /home/kevin/pdev/lmrl/language/language/search_agents/env/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py:1119: batch_scatter_update (from tensorflow.python.ops.state_ops) is deprecated and will be removed after 2018-11-29.
Instructions for updating:
Use the batch_scatter_update method of Variable instead.
2021-11-28 23:48:10.242020: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3299990000 Hz
2021-11-28 23:48:25.853691: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-11-28 23:48:26.521136: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-11-28 23:48:27.696041: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
```

I checked with nvidia-smi -l 1 and `python3` is using more of the memory of the GPU. There is xorg and gnome-shell using a little.

Then if I run strace on the process, I see:

```
strace: Process 4823 attached
futex(0x1add8378, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REA LTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
futex(0x1add8378, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
futex(0x1add8378, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
futex(0x1add8378, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
futex(0x1add8378, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
```

I'm not sure where to search for this class of issues.