[QUESTION] Chapter 10 RandomizedSearchCV returned best model having worse performance.
- Dominant language
- Jupyter Notebook
- Stars
- 30k
- Forks
- 13.1k
- PR merge metrics
- No merged PRs in 30d
Description
**Notebook:** 10_neural_nets_with_keras.ipynb
**Cell:** 102
**Problem:**
After `RandomizedSearchCV` using exactly the same hyperparameters as in the book I encounter the issue that it returns worse model.
## Most important:
```Python
model = rnd_search_cv.best_estimator_.model()
model.evaluate(X_test, y_test)
```
```
162/162 [==============================] - 0s 1ms/step - loss: 4.5746
4.57464599609375
```
Expected: around the best observed loss while `RandomizedSearchCV`
e.g.
```
Epoch 96/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2351 - val_loss: 0.2832
```
## **To Reproduce**
```Python
def build_model(n_hidden=1, n_neurons=30, learning_rate=3e-3, input_shape=[8]):
model = keras.models.Sequential()
options = {"input_shape": input_shape}
for layer in range(n_hidden):
model.add(keras.layers.Dense(n_neurons, activation="relu", **options))
options={}
model.add(keras.layers.Dense(1, **options))
optimizer = keras.optimizers.SGD(learning_rate=learning_rate)
model.compile(loss='mse', optimizer=optimizer)
return model
```
Now it is not possible to get keras.wrappers.KerasRegression, hence:
```Python
from scikeras.wrappers import KerasRegressor
keras_reg = KerasRegressor(model=build_model, learning_rate=None, n_hidden=None, n_neurons=None)
```
```Python
from scipy.stats import reciprocal
from sklearn.model_selection import RandomizedSearchCV
param_distribs = {
"n_hidden": [0, 1, 2, 3],
"n_neurons": np.arange(1, 100),
"learning_rate": reciprocal(3e-4, 3e-2),
}
rnd_search_cv = RandomizedSearchCV(keras_reg, param_distribs, n_iter=10, cv=3)
rnd_search_cv.fit(X_train, y_train, epochs=100,
validation_data=(X_valid, y_valid),
callbacks=[keras.callbacks.EarlyStopping(patience=10)])
```
A few of last lines:
```stacktrace
363/363 [==============================] - 1s 2ms/step - loss: 0.2377 - val_loss: 0.3169
Epoch 93/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2348 - val_loss: 0.2939
Epoch 94/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2333 - val_loss: 0.2923
Epoch 95/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2353 - val_loss: 0.2991
Epoch 96/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2351 - val_loss: 0.2832
Epoch 97/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2326 - val_loss: 0.2875
Epoch 98/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2324 - val_loss: 0.2996
Epoch 99/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2308 - val_loss: 0.3063
Epoch 100/100
363/363 [==============================] - 1s 2ms/step - loss: 0.2318 - val_loss: 0.2876
```
Best params:
```Python
rnd_search_cv.best_params_, rnd_search_cv.best_score_
```
```
({'learning_rate': 0.009847435984689484, 'n_hidden': 3, 'n_neurons': 74},
0.7897341518291179)
```
In the book:
- learning_rate=0.0058 (but differences expected)
- score = -0.32039451599121094
## Important debug information:
```Python
model.summary()
```
```
Model: "sequential_98"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_279 (Dense) (None, 30) 270
dense_280 (Dense) (None, 1) 31
=================================================================
Total params: 301 (1.18 KB)
Trainable params: 301 (1.18 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
```
**Expected behavior**
The best model found in the `RandomizedSearchCV` does not seem to be applied for the best model and the model is fixed
**Versions:**
- Windows 11
- Python: 3.11.4
- TensorFlow: 2.13.0
- keras: 2.13.1
- Scikit-Learn: 1.2.2
- scikeras: 0.11.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.