Quantco / Quantco/glum

When fitting a lasso, is normalization performed even if fit_intercept=False?

Open
#488 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
386
Forks
36
Avg merge
23h 24m
Merged PRs (30d)
2

Description

I'm trying to understand what's the stopping criterion on the (sub)gradient norm, in order to scale gradient_tol correctly.

If I understood correctly, for a quadratic datatif/family="gaussian", and vanishing coefs, the grad argument of _norm_min_subgrad should be X.T @ y / len(y)

When I debug, I see this being true only when I have previously centered and normalized X. otherwise, even if in my GeneralizedLinearRegressor I have set fit_intercept=False, I see:

  • a P1 array which is not constant
  • in update_quadratics(), gradient_rows is equal to y / len(y) as expected, however grad = gradient_rows @ data.X does not yield a grad equal to X.T @ y / len(y).
    It seems this is due to X being a
    Mat: <class 'tabmat.dense_matrix.DenseMatrix'> of shape (10, 5). Shift: [0. 0. 0. 0. 0.] Mult: [1.03404146 1.22010398 0.81522932 1.0686763 1.27198591] so scaling happens.

Why is this the case eventhough clf.scale_predictors, and clf._center_predictors are False ?

To reproduce, print gradient_rows and grad in update_quadratics, and run:

from glum import GeneralizedLinearRegressor
import numpy as np

np.random.seed(0)
X = np.random.randn(10, 5)
y = np.random.randn(10)
X -= X.mean(axis=0)
alpha = 0.001


clf = GeneralizedLinearRegressor(
    alpha=alpha, gradient_tol=1000, fit_intercept=False, family="gaussian",
    l1_ratio=1, verbose=10).fit(X, y)

print("grad rows should be", y / len(y))  # it is
print("grad should be", X.T @ y / len(y))   # it is not!

If you center and scale X before, then the problem disappears

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with GeneralizedLinearRegressor.fit and the update_quadratics() path mentioned in the report, then inspect _norm_min_subgrad and tabmat's DenseMatrix shift and multiplier values. Run the provided reproduction and compare the transformed X with the expected X.T @ y / len(y). Done means establishing whether scaling is intended when fit_intercept=False and documenting or correcting the behavior with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.