Slow h5repack performance on a large file
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
Hi,
I'm trying to repack the following file https://s3.dev.maayanlab.cloud/archs4/files/human_gene_v2.latest.h5 to change the chunk size of the `/data/expression` dataset.
```
$ h5ls -v human_gene_v2.latest.h5/data/expression
Opened "human_gene_v2.latest.h5" with sec2 driver.
expression Dataset {67186/67186, 1093742/Inf}
Location: 1:1832
Links: 1
Chunks: {2000, 1} 8000 bytes
Storage: 293936600048 logical bytes, 55971368226 allocated bytes, 525.16% utilization
Filter-0: deflate-1 OPT {4}
Type: native unsigned int
```
I've used the following command:
```
h5repack -i human_gene_v2.latest.h5 -o human_gene_v2.latest.repack.h5 -l "/data/expression:CHUNK=67186x10" -f "/data/expression:GZIP=1" -v
```
After 1.5 hours, it wasn't done at all, only showing 8 GB file, that is about 10-15% of the work done.
Is this expected behavior? Is there any standard way to make this process faster?
Claude Code made a h5py based script that process the whole file in the same ~1.5 hours. It says that the performance is due to `rdcc_nbytes` and `rdcc_nslots` parameters, but I'm not sure if it's true:
```{python}
src = h5py.File(args.input, "r",
rdcc_nbytes=cache_bytes, rdcc_nslots=1_000_003)
```
Can something like this be supported in the official h5repack, or getting a custom script like this is my best bet?
Thank you
Contributor guide
Assessment
This issue has not been assessed yet.