Option(s) to disable various kinds of external file access
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
HDF5 files have at least 3 ways by design to refer to data in other files: external datasets, virtual datasets & external links. In each case, by design, code reading data doesn't need to do anything extra: regular instructions to read from a dataset can transparently open a separate file. I think there's a fourth way in the 'multi' file driver, and maybe more I haven't noticed.
Many applications do not use any of these, and thus people reasonably see HDF5 as a data container format, and are unpleasantly surprised to discover the external reference features. E.g. [this HDF5 issue](https://github.com/h5py/h5py/issues/2891), and [this blog post](https://huggingface.co/blog/agent-intrusion-technical-timeline#vector-1-hdf5-external-raw-storage-file-read), where external datasets appear to have been involved in a real compromise. External datasets are particularly bad for this, because the target file doesn't need to be HDF5 format.
Would it be feasible to have switches to disable/enable these features? I would probably put this on the level of opening a file, so you can open a file and specify it should only access data within the file (or e.g. allow virtual datasets but not other mechanisms). Then attempting to read an external dataset would fail with an error.
Separately, what about forbidding certain parts of the filesystem, especially for external datasets? The blog post I linked above mentions credentials being exfiltrated from `/proc/self/environ`, and it's hard to see a legitimate use case for referring to files in `/proc`, `/sys` or `/dev` on Linux. This obviously doesn't eliminate the concern, but it could cut off a few ways to abuse it.
There are already APIs to set a prefix for looking up external files, but AFAIK this is just a starting point for relative paths, not a restriction on what can be opened. It might also be useful to allow setting a prefix so that only files within that can be referenced (although it probably needs some care with symlinks).
Contributor guide
Assessment
This issue has not been assessed yet.