Scan error "too many items"

Summary:

Please provide a short summary (no more than a sentence or two).

When extracting samples from estimated posteriors from many parameters a "too many items" error is thrown by Rstan.

Description:

Describe the issue as clearly as possible.

I have built a model that uses Dirichlet and multinomial distributions to model count data. The model works well on smaller datasets, but when the dimensionality of the distributions increases rstan has issues. Sampling of posteriors appears to have completed and the error seems to be related to scanning a temporary file created by Rstan. This issue has been previously documented in 2018 here:

Issue: https://github.com/stan-dev/rstan/issues/530
and in 2019 here:
https://discourse.mc-stan.org/t/problem-with-unserialize-with-reasonably-large-model/2484

Here is the error:
Error in scan(csvfile, what = double(), sep = ",", comment.char = "", :
too many items
Calls: cnvrg_VI ... -> -> .local -> read_one_stan_csv -> scan

This appears to be an open issue and the only solution posted is to patch base R. I am attempting to make a user friendly R package for R stan, so asking users to patch base R is not very practical. Solutions would be appreciated!

Reproducible Steps:

From issue #530
Generate a stan output CSV with more than INT_MAX total values, and try to load into stan using the read_stan_csv function.

RStan Version:

The version of RStan you are running (e.g., from packageVersion("rstan"))
Version 2.18.2, GitRev: 2e1f913d3ca3

R Version:

The version of R you are running (e.g., from R.version.string)
3.5.0

Operating System:

Your operating system (e.g., OS X 10.11.3)
Red Hat Enterprise Linux Server
VERSION="7.9 (Maipo)"

Thanks!

Start by reproducing the failure described in issue #530: generate a Stan output CSV with more than INT_MAX total values and load it with read_stan_csv. Inspect the scan call in the read_stan_csv path and compare the behavior with the linked reports. Done means large posterior output can be loaded without the “too many items” error.