scikit-learn / scikit-learn/scikit-learn

Document what cost function LogisticRegression minimizes for each solver

Open
#10,164 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Documentation module:linear_model
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

Description

Different choices of solver for sklearn's LogisticRegression optimize different cost functions - its highly confusing behavior, particularly of concern if you want to publish what cost function you're using. In particular:

sklearn.LR(solver=liblinear) minimizes: L + lam*Rb
sklearn.LR(solver=others) minimizes:    L + lam*R
statsmodels.GLM(bionomial) minimizes:   L/n + lam*Rb

where:

lam = 1/C
L = logloss
n = training sample size
R = square of L2 norm of feature weights
Rb =square of L2 norm of feature weights and intercept

I was a little surprised to find that the logloss is not normalized by the training set size. I think this is uncommon, and means the effective C changes based on the amount of training data. Good thing, bad thing? Not sure, but it seems unusual, but more importantly, what is minimized should be explicit.

PS. https://github.com/scikit-learn/scikit-learn/pull/10001 --- excellent idea! The default liblinear cost function is just plain confusing.

Steps/Code to Reproduce

There's an example to show the different weights here:

https://stackoverflow.com/questions/47338695/why-does-the-choice-of-solver-result-in-different-weight-in-sklearn-logisticreg

Expected Results
Actual Results
Versions

1.19.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LogisticRegression documentation and its solver descriptions, then compare the stated objectives for liblinear and the other solvers with the existing text. Done means the minimized cost function, regularization terms, intercept treatment, and normalization are explicitly documented for each solver.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.