benchopt / benchopt/benchmark_ridge

Ridge with GLMNET: discrepancies

Open
#2 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
5
PR merge metrics
No merged PRs in 30d

Description

`glmnet` ridge does not seem to always give the same results as others on simple datasets (even outside of Benchopt, that's why I present the R code):
To solve the ridge problem: $\min \frac{1}{2}||y-X\beta||_2^2 + \frac{\lambda}{2}||\beta||_2^2$ the penalty in `glmnet` is: $\lambda_R=\frac{\sigma(y)}{n}\lambda$ with $\sigma(y)$ the french standard deviation

```r
set.seed(123)
n <- 1000
p <- 100
X <- matrix(rnorm(n*p,0,1),n,p)
beta <- rnorm(p,0,1)
Y <- X%*%beta+rnorm(n,0,0.5)
sd_y <- sqrt(var(Y)*(n-1)/n)[1,1]
beta1 <- solve(t(X)%*%X+10*diag(p),t(X)%*%(Y))[,1]
fit_glmnet <- glmnet(X,Y, alpha=0.001, standardize = F, intercept = FALSE, thresh = 1e-20, lambda=sd_y*10/n)
beta2 <- as.vector(coef(fit_glmnet))[-1]
cbind(beta1[1:10], beta2[1:10]) # same beta
```

But on datasets like `bodyfat`:
```r
X <- bodyfat[, 4:17]
X <- sapply(X, as.numeric)
Y <- bodyfat$siri
p <- dim(X)[2]
n <- dim(X)[1]
sd_y <- sqrt(var(Y)*(n-1)/n)
beta1 <- solve(t(X)%*%X+10*diag(p),t(X)%*%(Y))
fit_glmnet <- glmnet(X,Y, alpha=0.001, standardize = F, lambda=sd_y*10/n,
intercept = F, thresh = 1e-20, maxit=10000000)
beta2 <- as.vector(coef(fit_glmnet))[-1]
cbind(beta1[1:10], beta2[1:10]) # not exactly same beta
```

@cassiofragadantas found [this post](https://stats.stackexchange.com/questions/74206/ridge-regression-results-different-in-using-lm-ridge-and-glmnet) that also says there are discrepancies not explained by rescaling
So how would we treat this in Benchopt ? (a case where the solver does not converge to the same coefficients as others sometimes)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.