HDF backend slow

General information:

• emcee version: 3.0.2
• platform: opensuse
• installation method (pip/conda/source/other?): pip

I am looking at your saving example at https://emcee.readthedocs.io/en/v3.0.2/tutorials/monitor/

Problem description:

I very much like the flexibility offered by the HDF backend, being able to save my chain to a file and to continue at a later point in time (especially as backup for long computations when my cpu node dies). However, when I have a fast log_prob function the overhead of opening/writing/closing the HDF file on each iteration seems to be disproportionately high and the overall computation performance is painfully slow.

Expected behavior:

Perhaps an easy solution would be an option to only save the chain state on every n-th iteration (where n is some adjustable number or n is calculated based on the relative progress). This may save some overhead by only opening/closing the HDF file once in a while.

Start with the v3.0.2 saving and monitor tutorial and trace how the HDF backend saves the chain on each iteration. Determine where an adjustable save interval would belong; done means long runs can save periodically while still allowing a saved chain to be resumed.