twosigma / twosigma/flint

Regression/summarizer performance collapse?

Open
#74 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.2k
Forks
199
PR merge metrics
No merged PRs in 30d

Description

Hello, I am using many regressions in parallel over a single call to `summarize`. I've noticed that if I run ~20 regressions on a dataset with 5M rows, it seems to take 45-60 minutes to summarize. If I run a single regression on a similarly-sized dataset, however, it only takes a minute or two to summarize. What kinds of performance characteristics should I expect, and how can I avoid this kind of performance collapse?

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.