HDFGroup / HDFGroup/hdf5

Option(s) to disable various kinds of external file access

Open
#6,618 4 comments 2 reactions 1 assignee Claimed by @vchoi-hdfgroup View on GitHub
Component - C Library
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

HDF5 files have at least 3 ways by design to refer to data in other files: external datasets, virtual datasets & external links. In each case, by design, code reading data doesn't need to do anything extra: regular instructions to read from a dataset can transparently open a separate file. I think there's a fourth way in the 'multi' file driver, and maybe more I haven't noticed.

Many applications do not use any of these, and thus people reasonably see HDF5 as a data container format, and are unpleasantly surprised to discover the external reference features. E.g. [this HDF5 issue](https://github.com/h5py/h5py/issues/2891), and [this blog post](https://huggingface.co/blog/agent-intrusion-technical-timeline#vector-1-hdf5-external-raw-storage-file-read), where external datasets appear to have been involved in a real compromise. External datasets are particularly bad for this, because the target file doesn't need to be HDF5 format.

Would it be feasible to have switches to disable/enable these features? I would probably put this on the level of opening a file, so you can open a file and specify it should only access data within the file (or e.g. allow virtual datasets but not other mechanisms). Then attempting to read an external dataset would fail with an error.

Separately, what about forbidding certain parts of the filesystem, especially for external datasets? The blog post I linked above mentions credentials being exfiltrated from `/proc/self/environ`, and it's hard to see a legitimate use case for referring to files in `/proc`, `/sys` or `/dev` on Linux. This obviously doesn't eliminate the concern, but it could cut off a few ways to abuse it.

There are already APIs to set a prefix for looking up external files, but AFAIK this is just a starting point for relative paths, not a restriction on what can be opened. It might also be useful to allow setting a prefix so that only files within that can be referenced (although it probably needs some care with symlinks).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.