20% performance regression in 1.1.2/1.1.3

Hello,

I believe there is a performance regression between 1.1.1 and 1.1.2

Simple reproducer
```python
import statistics
import timeit

import pandas as pd
import prophet

# https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv
#df_in = pd.read_csv("./example_wp_log_peyton_manning.csv")
df_in = pd.read_csv("https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv")

def run_prophet_speedtest(periods=30, include_history=False, holiday=True):
m = prophet.Prophet(mcmc_samples=0, uncertainty_samples=1000)
if holiday:
m.add_country_holidays(country_name="US")
m.fit(df_in)

# Python
future = m.make_future_dataframe(periods=periods, include_history=include_history)
forecast = m.predict(future)
return m, forecast

t = timeit.Timer(lambda: run_prophet_speedtest())

ret = t.repeat(repeat=10, number=6)
mean = statistics.mean(ret)
median = statistics.median(ret)
std = statistics.stdev(ret)

print(f"results: {ret}")
print(f"mean: {mean}, median: {median}, std: {std}")
```

try with:
```
pip install prophet==1.1.1
pip install prophet==1.1.2
```

I have also my own build on which I have cherry picked some commits, and I get a speed increase compared to 1.1.1, with this commit: https://github.com/facebook/prophet/commit/07167512883f2035c8f3b2dada2a64f2cf698bdf

I still need to bisect, but I suspect it's either the fourier related commits or this warm start, https://github.com/facebook/prophet/commit/eb71d0117b7154e2b584e11fd2b7ea7aa13a2128 . Any ideas?