joblib.parallel with patsy?

So one of my datasets can be super-huge and therefore, super-slow by my impatient standards. I can sit there for 3-5 minutes just waiting for a design matrix to be built (on one core). There's lots of terms/factors in this model. I have plenty of spare CPU's just sitting waiting for this to finish before the GLM can be kicked off.

Crazy idea... would it make any sense to incorporate joblib.parallel in patsy?

I have 2 ideas...

Simple idea is to just call joblib.parallel on the dmatrix call and make those in parallel.

Slightly more crazy idea would be to try to incorporate this inside patsy. I see 2 potential (somewhat) easy placements...

here and here

I recognize this brings opens a can of worms--transferring the data and state between the parent and child nodes is not free... new complexity in debugging... And doing things in parallel should be after other optimizations. (Tell me... is 5 minutes to run dmatrix on 30 million rows with about 50 output columns slower than you'd think?)

So before I go down this potentially very dumb rabbit hole, I'm happy to hear the thoughts of people who have better knowledge/experience with these topics.

Start by reading patsy/build.py at the two locations linked in the issue, lines 442 and 886, and review how the dmatrix call is involved. Benchmark the reported large-dataset case before evaluating joblib.parallel; the issue does not yet define an agreed implementation or testable completion criteria.