fsspec / fsspec/gcsfs

Can't open gcsfuse-mounted HDF5 file with h5py

Open
#107 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

First I ran:

gcsfuse <bucket> /tmp/<bucket>

Then, attempting to open an HDF5 file with h5py:

>>> from h5py import *
>>> input = '/tmp/<bucket>/file'
>>> f = File(input, 'r')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "…/venv/lib/python3.6/site-packages/h5py/_hl/files.py", line 312, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
  File "…/venv/lib/python3.6/site-packages/h5py/_hl/files.py", line 142, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 78, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

I'm not sure whether this is likely a bug in h5py or gcsfuse; I can see the file with e.g. ls or or.path.getsize, so gcsfuse seems to be doing its job, but my guess is that there is some syscall or access pattern in h5py that in not playing nicely with gcsfuse.

(mostly-irrelevant discussion of this error message in h5py)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the gcsfuse mount and h5py open sequence, starting with h5py/_hl/files.py and the h5py.h5f.open call shown in the traceback. Compare access to the same HDF5 file locally and determine whether the failure belongs to h5py or the mounted Google Cloud Storage filesystem; done should be a confirmed cause with an actionable fix or a clear upstream report.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.