Virtual datasets: control how errors are handled when opening source files
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
**Is your feature request related to a problem? Please describe.**
When reading data from a virtual dataset, errors opening the source files are silenced, and the corresponding portion of the data contains the fill value. There are various reasons that opening a file might fail:
- The source file has been moved/deleted
- The path is relative and the VDS file has been moved
- The user doesn't have permission to read the source file
- There's a problem with the filesystem, especially network filesystems (this happens occasionally for us)
Giving back incorrect data is, for our use cases, never desirable. We would much rather the entire read failed with the error showing what has gone wrong.
**Describe the solution you'd like**
Ideally, I'd like reads from VDSs to raise these errors by default, but from what I know of unlimited VDSs and printf-style source formatting, I suspect that's not on the table. So the next best thing would be an option you can set on a property list, something like `H5Pset_virtual_error_behaviour`. I can see 3 options:
- Silence all errors (current behaviour)
- Silence file not found errors (Unix `ENOENT`) but not others such as permission errors
- Propagate any error accessing the source files
**Describe alternatives you've considered**
- We have a script ([hdf5-vds-check](https://github.com/European-XFEL/hdf5-vds-check)) to check for these issues, but it's not used routinely, and the problem affects scientists who don't know about these internal details of HDF5.
- We could check before or after reading if the dataset is virtual and if we can open all the source files. But:
- There's a performance cost to doing an extra check
- This goes against the notion that virtual datasets are transparent, and you can use them like any other dataset
- Transient filesystem issues could mean there's an error when HDF5 tries to open a source file, but not when we do the separate check, so it's hard (impossible?) to make it completely reliable
- At the extremes, we could either abandon using virtual datasets entirely, or reimplement the machinery to read them to avoid this issue.
Contributor guide
Assessment
This issue has not been assessed yet.