google-deepmind / google-deepmind/deepmind-research
learn to simulate
- Dominant language
- Jupyter Notebook
- Stars
- 15.2k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to train a model using the fallowing command :
` python3.6 -m learning_to_simulate.train --data_path=./learning_to_simulate/tmp/datasets/WaterDrop/ --model_path=./learning_to_simulate/tmp/models/WaterDrop`
and I get the fallowing error :
2022-11-10 13:53:37.015640: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 37778160 exceeds 10% of system memory.
2022-11-10 13:53:37.084779: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "/home/soheilsh/anaconda3/envs/simulate/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/soheilsh/anaconda3/envs/simulate/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/soheilsh/anaconda3/envs/simulate/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(1356, 30), b.shape=(30, 128), m=1356, n=128, k=30
[[{{node EncodeProcessDecode/graph_independent/node_model/sequential/mlp/linear_0/MatMul}}]]
(1) Internal: Blas GEMM launch failed : a.shape=(1356, 30), b.shape=(30, 128), m=1356, n=128, k=30
[[{{node EncodeProcessDecode/graph_independent/node_model/sequential/mlp/linear_0/MatMul}}]]
[[truediv_4/_4847]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "anaconda3/envs/simulate/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
...
GPU : NVIDIA GeForce RTX 3060 Laptop
How can I solve this problem ?
Many thanks in advance for your response :)
Contributor guide
Research direction
Start by reproducing the `python3.6 -m learning_to_simulate.train` command with the provided WaterDrop data and model paths. Investigate the reported TensorFlow cuBLAS GEMM failure on the NVIDIA GeForce RTX 3060 Laptop and verify the CUDA/GPU environment. Done means the training command runs without the reported `CUBLAS_STATUS_EXECUTION_FAILED` error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100