H5Ocopy does not copy dimension scales properly
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
The [dimension scales](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_s__u_g.html#title8) system creates objects which can't be correctly copied by H5Ocopy. I've included some Python code below which demonstrates this - apologies for not taking the time to rewrite it in C, but I'm pretty confident this is an issue with HDF5 itself, not h5py. I've tested with HDF5 2.0.
The `DIMENSION_LIST` attribute stores a vlen array of references. However, the code to copy attributes only checks if the top-level type is H5T_REFERENCE, whereas here it will be H5T_VLEN. The code has a comment noting that it won't work for nested references.
https://github.com/HDFGroup/hdf5/blob/8cd9f7a7ba6757fbb72e36bbe23e127f8507c8a6/src/H5Aint.c#L2418-L2423
In isolation, nested references seems like a corner case that we could live without, but it's a shame that the dimension scale system got designed in a way that uses them.
Python reproducer
```python
import h5py
import numpy as np
# Set up a simple file with a dimension scale
f1 = h5py.File("dims1.h5", "w")
g1 = f1.create_group('g')
g1['data'] = np.ones((4, 5))
g1['x1'] = np.arange(4)
g1['x1'].make_scale()
g1['data'].dims[0].attach_scale(g1['x1'])
# Looking up the scale in f1 is OK
ref1 = f1['g/data'].attrs['DIMENSION_LIST'][0][0]
print("Use reference in file 1:", f1[ref1])
print("Use dimension scales in file 1:", f1['g/data'].dims[0].items())
# Open a new file and copy the group to it
f2 = h5py.File("dims2.h5", "w")
# This calls H5Ocopy with H5O_COPY_EXPAND_REFERENCE_FLAG
f1.copy('g', f2, 'g', expand_refs=True)
# Looking up the scale in f2 fails
ref2 = f2['g/data'].attrs['DIMENSION_LIST'][0][0]
print("Use reference in file 2:", f2[ref2])
print("Use dimension scales in file 2:", f2['g/data'].dims[0].items())
```
h5dump of the files created by the above code
```
HDF5 "dims1.h5" {
GROUP "/" {
GROUP "g" {
DATASET "data" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 4, 5 ) / ( 4, 5 ) }
DATA {
(0,0): 1, 1, 1, 1, 1,
(1,0): 1, 1, 1, 1, 1,
(2,0): 1, 1, 1, 1, 1,
(3,0): 1, 1, 1, 1, 1
}
ATTRIBUTE "DIMENSION_LIST" {
DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }
DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
DATA {
(0): (DATASET 94797652731280 "/g/x1"), ()
}
}
}
DATASET "x1" {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
DATA {
(0): 0, 1, 2, 3
}
ATTRIBUTE "CLASS" {
DATATYPE H5T_STRING {
STRSIZE 16;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "DIMENSION_SCALE"
}
}
ATTRIBUTE "NAME" {
DATATYPE H5T_STRING {
STRSIZE 1;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): ""
}
}
ATTRIBUTE "REFERENCE_LIST" {
DATATYPE H5T_COMPOUND {
H5T_REFERENCE { H5T_STD_REF_OBJECT } "dataset";
H5T_STD_U32LE "dimension";
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): {
DATASET 94797652631664 "/g/data",
0
}
}
}
}
}
}
}
HDF5 "dims2.h5" {
GROUP "/" {
GROUP "g" {
DATASET "data" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 4, 5 ) / ( 4, 5 ) }
DATA {
(0,0): 1, 1, 1, 1, 1,
(1,0): 1, 1, 1, 1, 1,
(2,0): 1, 1, 1, 1, 1,
(3,0): 1, 1, 1, 1, 1
}
ATTRIBUTE "DIMENSION_LIST" {
DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }
DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
DATA {
(0): (), ()
}
}
}
DATASET "x1" {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
DATA {
(0): 0, 1, 2, 3
}
ATTRIBUTE "CLASS" {
DATATYPE H5T_STRING {
STRSIZE 16;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "DIMENSION_SCALE"
}
}
ATTRIBUTE "NAME" {
DATATYPE H5T_STRING {
STRSIZE 1;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): ""
}
}
ATTRIBUTE "REFERENCE_LIST" {
DATATYPE H5T_COMPOUND {
H5T_REFERENCE { H5T_STD_REF_OBJECT } "dataset";
H5T_STD_U32LE "dimension";
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): {
,
0
}
}
}
}
}
}
}
```
Contributor guide
Assessment
This issue has not been assessed yet.