ageron / ageron/handson-ml3

[QUESTION] Chapter 4 Exercise Question 12 - cost function with l2 regularization seems incorrect

Open
#118 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
14.1k
Forks
5.3k
PR merge metrics
No merged PRs in 30d

Description

When attempting the question, there is a bonus part to add l2 regularization to the softmax regression code (`In [75]`):
![image](https://github.com/ageron/handson-ml3/assets/82577844/0478efb5-71d2-4fdf-8fdc-30fde6f1c922)

According to the book, in the section about Ridge Regression, we are supposed to add ($\dfrac{\alpha}{m}$ * sum of thetas) to the original cost function. However in line 2 in the picture above, `l2_loss` is somehow calculated with 1/2 multiplied at the front. Shouldn't it be 1/m instead?

![image](https://github.com/ageron/handson-ml3/assets/82577844/f582caee-a93b-4082-8c67-f52197b81294)
According to the same section of the book, we should add $2\alpha w / m$ to the MSE gradient vector. So in line 3 of the picture above, shouldn't it be `2 * alpha * Theta[1:] / m` instead?

Maybe this is why the validation loss suddenly increased a lot when the regularization is applied.

If this is indeed a typo, the bottom sections involving the hyperparameter `C` also has to be changed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.