Usage of ancient files for testing breaks with recent h5py/libhdf5
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 104
- Forks
- 28
- Avg merge
- 22h 47m
- Merged PRs (30d)
- 6
Description
While working on #102 @bnlawrence and I found a glitch when we recreated test files with recent h5py/libhdf5.
See https://github.com/NCAS-CMS/pyfive/pull/102#discussion_r2393563713 for details.
Citing here for visibility:
If we recreate the file now, the following line of code:
attrs['vlen_str_array'] = [b'Hello', b'World!']
translates to this HDF5 dump:
ATTRIBUTE "vlen_str_array" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
DATA {
(0): "Hello", "World!"
}
}
whereas the old file translates to:
ATTRIBUTE "vlen_str_array" {
DATATYPE H5T_STRING {
STRSIZE 6;
STRPAD H5T_STR_NULLPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
DATA {
(0): "Hello\000", "World!"
}
}
It might make sense to move from the current behaviour of testing already created files (and ingesting them into GIT) to creating test files only when testing in the test suite. This is what updownstream libraries like xarray and h5netcdf are doing. Thoughts?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the test suite and the ancient HDF5 files discussed in pull request #102, then read the linked discussion about h5py/libhdf5 attribute encoding. Compare the current checked-in-file approach with the on-demand test-file approach used by xarray and h5netcdf. Done means the affected tests work with recent h5py/libhdf5 without relying on incompatible recreated files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100