NCAS-CMS / NCAS-CMS/pyfive

Usage of ancient files for testing breaks with recent h5py/libhdf5

Open
#108 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
104
Forks
28
Avg merge
22h 47m
Merged PRs (30d)
6

Description

While working on #102 @bnlawrence and I found a glitch when we recreated test files with recent h5py/libhdf5.

See https://github.com/NCAS-CMS/pyfive/pull/102#discussion_r2393563713 for details.

Citing here for visibility:

If we recreate the file now, the following line of code:

attrs['vlen_str_array'] = [b'Hello', b'World!']

translates to this HDF5 dump:

ATTRIBUTE "vlen_str_array" {
   DATATYPE  H5T_STRING {
      STRSIZE H5T_VARIABLE;
      STRPAD H5T_STR_NULLTERM;
      CSET H5T_CSET_ASCII;
      CTYPE H5T_C_S1;
   }
   DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
   DATA {
   (0): "Hello", "World!"
   }
}

whereas the old file translates to:

ATTRIBUTE "vlen_str_array" {
   DATATYPE  H5T_STRING {
      STRSIZE 6;
      STRPAD H5T_STR_NULLPAD;
      CSET H5T_CSET_ASCII;
      CTYPE H5T_C_S1;
   }
   DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
   DATA {
   (0): "Hello\000", "World!"
   }
}

It might make sense to move from the current behaviour of testing already created files (and ingesting them into GIT) to creating test files only when testing in the test suite. This is what updownstream libraries like xarray and h5netcdf are doing. Thoughts?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the test suite and the ancient HDF5 files discussed in pull request #102, then read the linked discussion about h5py/libhdf5 attribute encoding. Compare the current checked-in-file approach with the on-demand test-file approach used by xarray and h5netcdf. Done means the affected tests work with recent h5py/libhdf5 without relying on incompatible recreated files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.