Some computations not included in reported total time of output CSVs

Summary:

Stan writes to the output csv file of each chain something like

#  Elapsed Time: 
#                0.008 seconds (Warm-up)
#                0.004 seconds (Sampling)
#                0.012 seconds (Total)

but this total time doesn't include all computations that have to be done with the model.

Description:

If we look at run_adaptive_sampler, we see that the times warm_delta_t and sample_delta_t measure just the runtime of util::generate_transitions() for warmup and sampling phases, respectively. Then they are written with writer.write_timing(warm_delta_t, sample_delta_t), meaning that the reported total time is their sum. So for example for hmc_static_diag_e_adapt() things that don't get included in the total time are:

A) Finding the initial starting point if not given
B) Calling sampler.init_stepsize() in run_adaptive_sampler()

There are also other things such as writing the CSV header and reading input but I listed these because these both include log probability and gradient evaluations with the model and therefore I think should be included in reported to measure model performance.

A) involves log probability and gradient evaluations until a point where they are finite is found (max 100 tries)
B) involves gradient evaluations and leapfrog steps inside a while loop until some condition is satisfied (https://github.com/stan-dev/stan/blob/develop/src/stan/mcmc/hmc/base_hmc.hpp)

I found this because I have ODE models where sampler.init_stepsize() can require millions of evaluations of the ODE function (confirmed with profiling) and therefore takes a substantial proportion of the actual wall time. I have sent an example to @rok-cesnovar.

Interfaces:

In cmdstanr, there is also a Sys.time() timing around the entire program call, so that measures the actual wall time correctly. The
$time() method of a fit object reports something like

$total
[1] 210.9202

$chains
  chain_id warmup sampling total
1        1  0.007    0.004 0.011
2        2  0.012    0.004 0.016

So the grand total time is correct, but total time of each chain doesn't involve the initialization of parameter values and step size.

Possible solutions

I am not sure what is the purpose of B) and why it is called even if step size if specified by user. To me this sounds like this is sort of warmup and could either be included in warmup time or then the reported time could have a separate "initialization" part that is included in total time of the chain.

Current Version:

v2.28.1

Start in src/stan/services/util/run_adaptive_sampler.hpp and trace how warm_delta_t, sample_delta_t, and writer.write_timing() are used. Read the initialization path in hmc_static_diag_e_adapt() and the step-size logic in src/stan/mcmc/hmc/base_hmc.hpp. Done means the per-chain timing and reported total account for the identified initialization computations, with the timing behavior validated.