cmdstanpy 1.0.2 potentially logging warnings too often?

Summary

I'm wondering if this change in 1.0.2:

CmdStanPy now computes some diagnostics after running HMC and will warn you about post-warmup divergences and treedepth exceptions

leads to unnecessary warnings.

Description:

After running an MCMC, this is the output of .diagnose():

Processing csv files: /var/folders/3x/ypx818511wsdnxc3knn7yy1r0000gn/T/tmpj9urc8se/prophet_modela2q2ppg4/prophet_model-20220629231921_1.csv, /var/folders/3x/ypx818511wsdnxc3knn7yy1r0000gn/T/tmpj9urc8se/prophet_modela2q2ppg4/prophet_model-20220629231921_2.csv, /var/folders/3x/ypx818511wsdnxc3knn7yy1r0000gn/T/tmpj9urc8se/prophet_modela2q2ppg4/prophet_model-20220629231921_3.csv, /var/folders/3x/ypx818511wsdnxc3knn7yy1r0000gn/T/tmpj9urc8se/prophet_modela2q2ppg4/prophet_model-20220629231921_4.csv

Checking sampler transitions treedepth.
21 of 600 (3.5%) transitions hit the maximum treedepth limit of 10, or 2^10 leapfrog steps.
Trajectories that are prematurely terminated due to this limit will result in slow exploration.
For optimal performance, increase this limit.

Checking sampler transitions for divergences.
No divergent transitions found.

Checking E-BFMI - sampler transitions HMC potential energy.
E-BFMI satisfactory for all transitions.

Effective sample size satisfactory.

Split R-hat values satisfactory all parameters.

Processing complete.

Overall it seems like the sampling was successful. However:

1. cmdstanpy logged a warning at the end of the sampling:

23:23:47 - cmdstanpy - WARNING - Some chains may have failed to converge.
	Chain 1 had 5 iterations at max treedepth (3.3%)
	Chain 2 had 1 iterations at max treedepth (0.7%)
	Chain 3 had 13 iterations at max treedepth (8.7%)
	Chain 4 had 2 iterations at max treedepth (1.3%)

I'm wondering what the thresholds are for logging the non-convergence warning and whether they're too loose?

2. I also get these warnings, although they might be specific to the model we're running with Prophet:

23:23:47 - cmdstanpy - WARNING - Non-fatal error during sampling:
Exception: normal_id_glm_lpdf: Scale vector is 0, but must be positive finite! (in '/Users/runner/work/prophet/prophet/python/stan/prophet.stan', line 137, column 2 to line 142, column 4)
Exception: normal_id_glm_lpdf: Scale vector is 0, but must be positive finite! (in '/Users/runner/work/prophet/prophet/python/stan/prophet.stan', line 137, column 2 to line 142, column 4)
Exception: normal_id_glm_lpdf: Matrix of independent variables is inf, but must be finite! (in '/Users/runner/work/prophet/prophet/python/stan/prophet.stan', line 137, column 2 to line 142, column 4)

It seems to be related to a type check of the stan program? I see this warning every time we do MCMC sampling now, even if overall the sampling procedure has no issues. Not sure where it's coming from -- ideally if it doesn't cause any sampling issues we wouldn't show it (in my opinion), but keen to get your thoughts.

In cmdstanpy==1.0.1, neither of these warnings were raised.

Thank you!!!

To reproduce, in case you need to:

from prophet import Prophet
import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/tcuongd/prophet/tcuongd-covid-notebook/examples/example_pedestrians.csv')
m = Prophet(mcmc_samples=300)
m.fit(df, show_progress=False)

Current Version:

cmdstanpy==1.0.2

Start by running the provided Prophet reproduction with cmdstanpy 1.0.2, then compare its warning behavior with 1.0.1. Trace how sampling diagnostics and non-fatal Stan errors are reported; done means determining whether the thresholds and repeated warnings are appropriate and defining the expected behavior.