Background/asynchronous sampling (feature request & design discussion)

I propose modification of the various run methods to enable doing so in the background rather than occupying the foreground R session. This is useful for scenarios where one wants to do other things without launching a new session, as well as to make runs robust to crashes of the foreground R session.

To achieve this, I propose separating out the initiation and polling of the processx processes.

To enable continued running amidst crashes of the foreground R session, I propose starting the processes with cleanup=FALSE and saving the PID in an RDS file so that ps can be used to resume management of the process. This also requires that files be used for stdout and stderr (rather than the processx default of using pollable pipes back to the R session) so that monitoring progress can resume by simply checking these files. This latter in turn calls for a change in the cmdstanr approach to polling from using the build-in processx polling to polling based on checking/parsing the stdout/stderr files (using mod time or filesize to inform whether to bother looking for newlines).

While not necessary for the above, I propose storing these files in a folder called stan_scratch (whose default-but-configurable location is the current working dir). (n.b. said folder is involved in the proposed implementations of these FRs as well: During-sampling diagnostics, Recompile only on changes to output of stanc3 auto-formatter )

Similarly, I propose starting a separate process for each chain (in parallel if parallel chains are requested). (I'm pretty sure that's what cmdstanr does already, but wanted to be explicit here)

Start by locating the various run methods and reviewing how processx processes are initiated and polled. Read the linked processx and ps discussion first; the design is complete only when background chains, persisted process state, and file-based output polling work across a foreground R-session crash.