Chapter 16: Training with GPU fails
- Dominant language
- Jupyter Notebook
- Stars
- 30k
- Forks
- 13.1k
- PR merge metrics
- No merged PRs in 30d
Description
Hi to all,
I simply ran the solution of exercise 8 but I got:
```
InvalidArgumentError: [_Derived_] Invalid input_h shape: [1,32,30] [19,32,30]
[[{{node CudnnRNNV3}}]]
[[sequential/gru/StatefulPartitionedCall]] [Op:__inference_train_function_3493]
Function call stack:
train_function -> train_function -> train_function
```
I suppose that is related to the usage of GPU because without GPU everything works.
At this point, I tried to recreate the solution from zero, but I got a similar error:
```
UnknownError: [_Derived_] CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond_29/then/_0/CudnnRNNV3}}]]
[[model_1/lstm_1/StatefulPartitionedCall]] [Op:__inference_distributed_function_5148492]
Function call stack:
distributed_function -> distributed_function -> distributed_function
```
In this case, removing `keras.layers.Masking()(x)` that I added everything works.
The full code of the second experiment is the following:
```
input = keras.layers.Input(shape=[max_length])
x = keras.layers.Reshape((-1, 1))(input)
#x = keras.layers.Masking()(x)
x = keras.layers.LSTM(15, return_sequences=True, activation="relu", kernel_initializer=tf.keras.initializers.GlorotNormal(seed=42))(x)
x = keras.layers.LSTM(5, activation="tanh", kernel_initializer=tf.keras.initializers.GlorotNormal(seed=42))(x)
out = keras.layers.Dense(2, activation="softmax", kernel_initializer=tf.keras.initializers.GlorotNormal(seed=42))(x)
model = keras.Model(inputs=[input], outputs=[out])
```
The version of TF is 2.2.0 by default, but I tried also with 2.1.0.
Do you have any idea of why these errors happen?
Thank you
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing Chapter 16 exercise 8 with TensorFlow 2.1.0 and 2.2.0 on a GPU, using the reported LSTM/GRU code and the keras.layers.Masking() line. Compare the GPU and non-GPU runs and determine whether the failure is tied to masking or the cuDNN path; done means the cause and a reliable resolution are identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- keras, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100