HDF backend file size

General information:

• emcee version: 3.1.3
• platform: Ubuntu 22.04.1 LTS
• installation method (pip/conda/source/other?): pip

Problem description:

The HDF backend file seems to shrink in size when copied and saved using the h5py module.

Expected behavior: The copied file size should be the same as the original file size.

Actual behavior: The copied file size is about half of the original file size.

What have you tried so far?:

Minimal example:

from os import stat, remove

import emcee
import h5py
import numpy as np

emcee.__version__


def log_prob(x, ivar):
    return -0.5 * np.sum(ivar * x ** 2)

ndim, nwalkers = 5, 100
ivar = 1. / np.random.rand(ndim)
p0 = np.random.randn(nwalkers, ndim)

# Set up the backend
# Don't forget to clear it in case the file already exists
filename = "test_hdf5/tutorial.h5"
remove(filename)
backend = emcee.backends.HDFBackend(filename)
backend.reset(nwalkers, ndim)

sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, args=[ivar], backend=backend)
sampler.run_mcmc(p0, 100)

f_src=h5py.File(filename, 'r')
file_size_KB=stat(filename).st_size >> 10
print('{} KB'.format(file_size_KB))

with h5py.File(filename, "r") as f:
    print(list(f['mcmc'].keys()))
    print(f['mcmc']['chain'])

new_file='test_hdf5/new_tutorial.h5'
remove(new_file)
f_dst=h5py.File(new_file,'w')
f_src.copy('mcmc',f_dst)
f_src.close()
f_dst.close()

file_size_KB=stat(new_file).st_size >> 10
print('{} KB'.format(file_size_KB))

with h5py.File(new_file, "r") as f:
    print(list(f['mcmc'].keys()))
    print(f['mcmc']['chain'])

test_hdf5.md

Start with the HDFBackend and the linked test_hdf5.md example, then reproduce the file copy using h5py while comparing the original and copied files. Inspect the backend output and the h5py copy behavior; done means the reported size discrepancy is explained and the expected copied-file behavior is covered by a verified result or test.