ageron / ageron/handson-ml2

[BUG] Chapter 15 naive prediction error with correction

Aperta
#421 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Jupyter Notebook
Stelle
30k
Fork
13.1k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

The naive prediction in Chapter 15 reproduced below is incorrect. It also raises a visual red flag because the error from the network was about 10x smaller (about 0.027). Quoting from Chapter 15, https://github.com/ageron/handson-ml2/blob/master/15_processing_sequences_using_rnns_and_cnns.ipynb
```
Let's compare this performance with some baselines: naive predictions and a simple linear model:

In [31]:
Y_naive_pred = Y_valid[:, -1:]
np.mean(keras.metrics.mean_squared_error(Y_valid, Y_naive_pred))
Out[31]:
0.22278848
```

An accurate way to calculate the naive prediction error would be as follows. The naive error is actually smaller than the RNN walk forward error. This makes sense because it is actually very hard to beat the naive prediction, even when walking forward with real data as the notebook did. (For example, 70% of the naive error would be excellent. 10% of the naive error is ludicrous.)
```
actual_Y_naive=np.array([np.concatenate([x,y]) for x,y in zip(X_valid[:,-1],Y_valid[:,:-1])])
print(np.average(np.square(Y_valid-actual_Y_naive)))
np.mean(keras.metrics.mean_squared_error(Y_valid,actual_Y_naive))
0.01524985
0.015249849
```

The main problem with the code in the existing notebook is that it is not dimensionally consistent. Here are some accurate ways to calculate subsets of the naive error to make it easier to see.

This is just using the last validation value vs. the 2nd to last validation value.
```
Y_naive_pred = Y_valid[:, -2]
np.average(np.square(Y_valid[:,0]-X_valid[:,-1,0]))
np.mean(keras.metrics.mean_squared_error(Y_valid[:,-1], Y_naive_pred))
0.0152704455
0.0152704455
```

This is just using the first validation value vs. the last value in the input.
```
Y_naive_pred=X_valid[:,-1,0]
print(np.average(np.square(Y_valid[:,0]-X_valid[:,-1,0])))
np.mean(keras.metrics.mean_squared_error(Y_valid[:,0],Y_naive_pred))
0.015137789
0.015137789
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.