ageron / ageron/handson-ml2

Chapter 16: Training with GPU fails

Open
#196 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
30k
Forks
13.1k
PR merge metrics
No merged PRs in 30d

Description

Hi to all,

I simply ran the solution of exercise 8 but I got:

```
InvalidArgumentError: [_Derived_] Invalid input_h shape: [1,32,30] [19,32,30]
[[{{node CudnnRNNV3}}]]
[[sequential/gru/StatefulPartitionedCall]] [Op:__inference_train_function_3493]

Function call stack:
train_function -> train_function -> train_function
```

I suppose that is related to the usage of GPU because without GPU everything works.
At this point, I tried to recreate the solution from zero, but I got a similar error:

```
UnknownError: [_Derived_] CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond_29/then/_0/CudnnRNNV3}}]]
[[model_1/lstm_1/StatefulPartitionedCall]] [Op:__inference_distributed_function_5148492]

Function call stack:
distributed_function -> distributed_function -> distributed_function
```

In this case, removing `keras.layers.Masking()(x)` that I added everything works.
The full code of the second experiment is the following:

```
input = keras.layers.Input(shape=[max_length])
x = keras.layers.Reshape((-1, 1))(input)

#x = keras.layers.Masking()(x)
x = keras.layers.LSTM(15, return_sequences=True, activation="relu", kernel_initializer=tf.keras.initializers.GlorotNormal(seed=42))(x)
x = keras.layers.LSTM(5, activation="tanh", kernel_initializer=tf.keras.initializers.GlorotNormal(seed=42))(x)
out = keras.layers.Dense(2, activation="softmax", kernel_initializer=tf.keras.initializers.GlorotNormal(seed=42))(x)

model = keras.Model(inputs=[input], outputs=[out])
```

The version of TF is 2.2.0 by default, but I tried also with 2.1.0.

Do you have any idea of why these errors happen?
Thank you

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing Chapter 16 exercise 8 with TensorFlow 2.1.0 and 2.2.0 on a GPU, using the reported LSTM/GRU code and the keras.layers.Masking() line. Compare the GPU and non-GPU runs and determine whether the failure is tied to masking or the cuDNN path; done means the cause and a reliable resolution are identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
keras, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.