HDFGroup / HDFGroup/hdf5

Slow h5repack performance on a large file

Open
#6,458 3 comments 0 reactions 1 assignee Claimed by @jhendersonHDF View on GitHub
Component - Tools
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

Hi,

I'm trying to repack the following file https://s3.dev.maayanlab.cloud/archs4/files/human_gene_v2.latest.h5 to change the chunk size of the `/data/expression` dataset.

```
$ h5ls -v human_gene_v2.latest.h5/data/expression
Opened "human_gene_v2.latest.h5" with sec2 driver.
expression Dataset {67186/67186, 1093742/Inf}
Location: 1:1832
Links: 1
Chunks: {2000, 1} 8000 bytes
Storage: 293936600048 logical bytes, 55971368226 allocated bytes, 525.16% utilization
Filter-0: deflate-1 OPT {4}
Type: native unsigned int
```

I've used the following command:
```
h5repack -i human_gene_v2.latest.h5 -o human_gene_v2.latest.repack.h5 -l "/data/expression:CHUNK=67186x10" -f "/data/expression:GZIP=1" -v
```

After 1.5 hours, it wasn't done at all, only showing 8 GB file, that is about 10-15% of the work done.

Is this expected behavior? Is there any standard way to make this process faster?

Claude Code made a h5py based script that process the whole file in the same ~1.5 hours. It says that the performance is due to `rdcc_nbytes` and `rdcc_nslots` parameters, but I'm not sure if it's true:
```{python}
src = h5py.File(args.input, "r",
rdcc_nbytes=cache_bytes, rdcc_nslots=1_000_003)
```

Can something like this be supported in the official h5repack, or getting a custom script like this is my best bet?

Thank you

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.