ageron / ageron/handson-ml2

Chapter 4: Ridge regression using Gradient Descent [QUESTION]

Open
#510 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
30k
Forks
13.1k
PR merge metrics
No merged PRs in 30d

Description

**Ridge regression using Gradient Descent**
Hi!
I was trying to implement a Ridge regression in gradient descent by adding alpha*theta to the MSE gradient vector (where theta is the parameter vector).

So I've used the following code:

```python
eta = 0.1 #learning rate
n_iterations = 1000
m=76
alpha = 1

theta_ridge = np.random.randn(2,1) #random starting theta
for iteration in range(n_iterations):
gradients = 2/m*X_b.T.dot(X_b.dot(theta_ridge)-y) + alpha*theta_ridge[1] #theta_ridge[1] in order to exclude the intercept (theta_ridge[0])
theta_ridge = theta_ridge - eta*gradients

theta_ridge
>> array([[-0.09590297],
[ 0.19180594]])
```

**Expected behavior**
I would expect the final parameters vector to be the same as the one that I find by using the Ridge module from sklearn.linear_model, but it is not:

```python
from sklearn.linear_model import Ridge
ridge_reg = Ridge(alpha=1, solver='lsqr')
ridge_reg.fit(X_b, y)

ridge_reg.coef_
>> array([[0. , 0.28397242]])
```
I've also tried different solvers (even Cholesky as reported in the book) but I always get different results.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.