HDFGroup / HDFGroup/hdf5

Inconsistent behaviour with variable-length strings and character encoding

Open
#544 6 comments 0 reactions 1 assignee Claimed by @gheber View on GitHub
Component - C Library
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

We have diverse HDF5 files that are utilized in our scientific applications for different purposes. These files mix different kinds of datasets, like strings, arrays of floating point values, and more.

However, quite recently, a colleague detected an unexpected behaviour with one of the files. If the code was requesting a certain dataset with variable-length strings, an HDF5 error appeared and the dataset could not be retrieved:

```
#000: ../../src/H5Dio.c line 179 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: ../../src/H5Dio.c line 435 in H5D__read(): unable to set up type info
major: Dataset
minor: Unable to initialize object
#002: ../../src/H5Dio.c line 936 in H5D__typeinfo_init(): unable to convert between src and dest datatype
major: Dataset
minor: Feature is unsupported
#003: ../../src/H5T.c line 4532 in H5T_path_find(): no appropriate function for conversion path
major: Datatype
minor: Unable to initialize object
```

After further investigation, it turns out that the character set of the strings in the dataset was set to ASCII, while we were asking HDF5 to read UTF8 strings instead (i.e., our file was not generated appropriately). Nonetheless, the problem is that we have not had hit this issue until now, and it does not reproduce as long as you read other similar datasets first.

**To summarize our findings**. It turns out that if you ask HDF5 to read a variable-length strings dataset with the correct encoding, you can then freely ask HDF5 to read the same or another dataset with a different encoding, even if it does not match the content of the file.

**Is this an expected behaviour, or are we missing something?** To illustrate the issue, I'm attaching below a small source code example that mimics our problem. You can compile it and execute the code in two different ways.

- To reproduce the error-free behaviour that reads first the string with the character set defined in the file (ASCII) and then with another one (UTF8), you can run the code directly with `./h5cset_error.out`.
- To reproduce the error we were facing and described above, add at least one argument to the executable, like `./h5cset_error.out test`.

For further reference, the issue reproduces on HDF5 v1.10.7, v1.12.0, and compiling manually HDF5 with the most recent commit available on this repository. We have tested it with GCC 9.3.0 + HPE-MPI v2.22, but no MPI file access property was given (i.e., using H5P_DEFAULT to avoid MPI-IO).

Thank you very much for your help and my apologies in advance if this is not an issue.

### Source Code to reproduce issue
Note: Error checking is ignored to keep the example footprint minimum.
```C
#include "hdf5.h"
#include
#include

#define FILE "test.h5"
#define DATASET "MyRandomDataset"

void write_dataset()
{
char* wdata[1] = { "Hello there!" };
hid_t file, filetype, space, dset;
hsize_t dims = 1;

// Create a new file using the default properties
file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

// Create the reference datatype with ASCII encoding
filetype = H5Tcopy(H5T_C_S1);
H5Tset_size(filetype, H5T_VARIABLE);
H5Tset_cset(filetype, H5T_CSET_ASCII);

// Create dataset with a simple dataspace
space = H5Screate_simple(1, &dims, NULL);
dset = H5Dcreate(file, DATASET, filetype, space, H5P_DEFAULT,
H5P_DEFAULT, H5P_DEFAULT);

// Write the string data to it
H5Dwrite(dset, filetype, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata);

// Close and release resources
H5Dclose(dset);
H5Sclose(space);
H5Tclose(filetype);
H5Fclose(file);
}

void read_dataset(int8_t enable_error)
{
char* rdata[1] = { NULL };
hid_t file, memtype, space, dset;
H5T_cset_t cset;
int test;

// Open the reference file
file = H5Fopen(FILE, H5F_ACC_RDONLY, H5P_DEFAULT);

for (test = 0; test < 2; test++)
{
// Open the dataset
dset = H5Dopen(file, DATASET, H5P_DEFAULT);

// Create the memory datatype
memtype = H5Tcopy(H5T_C_S1);
H5Tset_size(memtype, H5T_VARIABLE);

// Set the character set according to the test behaviour
if (enable_error)
{
// FAILS: The character set of the first read does NOT match
cset = (test == 0) ? H5T_CSET_UTF8 : H5T_CSET_ASCII;
}
else
{
// WORKS: The character set of the first read matches
cset = (test == 0) ? H5T_CSET_ASCII : H5T_CSET_UTF8;
}
H5Tset_cset(memtype, cset);

// Read the data from the file
H5Dread(dset, memtype, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata);

printf("Read #%d (cset=%s): '%s'\n",
test, (cset == H5T_CSET_ASCII ? "ASCII" : "UTF8"), rdata[0]);

// Close and release resources
space = H5Dget_space(dset);
H5Dvlen_reclaim(memtype, space, H5P_DEFAULT, rdata);
H5Sclose(space);
H5Tclose(memtype);
H5Dclose(dset);
}

H5Fclose(file);
}

int main(int argc, char **argv)
{
// Create the file and store the string
write_dataset();

// Read the string from the file
read_dataset(argc > 1);

return 0;
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.