scikit-learn / scikit-learn/scikit-learn
Document what cost function LogisticRegression minimizes for each solver
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
Description
Different choices of solver for sklearn's LogisticRegression optimize different cost functions - its highly confusing behavior, particularly of concern if you want to publish what cost function you're using. In particular:
sklearn.LR(solver=liblinear) minimizes: L + lam*Rb
sklearn.LR(solver=others) minimizes: L + lam*R
statsmodels.GLM(bionomial) minimizes: L/n + lam*Rb
where:
lam = 1/C
L = logloss
n = training sample size
R = square of L2 norm of feature weights
Rb =square of L2 norm of feature weights and intercept
I was a little surprised to find that the logloss is not normalized by the training set size. I think this is uncommon, and means the effective C changes based on the amount of training data. Good thing, bad thing? Not sure, but it seems unusual, but more importantly, what is minimized should be explicit.
PS. https://github.com/scikit-learn/scikit-learn/pull/10001 --- excellent idea! The default liblinear cost function is just plain confusing.
Steps/Code to Reproduce
There's an example to show the different weights here:
Expected Results
Actual Results
Versions
1.19.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the LogisticRegression documentation and its solver descriptions, then compare the stated objectives for liblinear and the other solvers with the existing text. Done means the minimized cost function, regularization terms, intercept treatment, and normalization are explicitly documented for each solver.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100