Quantco / Quantco/glum

[Medium] Fast solvers for situations with many columns.

Open
#133 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

on hold performance
Dominant language
Python
Stars
386
Forks
36
Avg merge
23h 24m
Merged PRs (30d)
2

Description

Currently, the solvers all build a Fisher matrix/Hessian. If the data matrix is n x m, this Fisher matrix will have shape m x m. If m is less than a few thousand, this is fine. However, if m gets very large, possibly due to lots of categorical variables, the current solvers will be inadequate because they are constructing a very large dense matrix.

Currently, we can avoid that matrix construction using the diag_fisher=True option. However, this is not particularly well supported and performance might not be very good. We should consider adding a benchmark with more than 10,000 columns to better track performance in these cases.

As part of this effort, it would be nice to separate the diag_fisher behavior and simply use a different solver function for high dimensional problems.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the current solver functions and the diag_fisher=True path described in the issue. Measure performance and memory use on data with more than 10,000 columns, then determine how a separate high-dimensional solver should be exposed and benchmarked. Done means high-column problems avoid constructing a dense m x m Fisher matrix and have performance coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.