fsspec / fsspec/kerchunk

Issue kerchunking NWM medium range netcdf via HTTPS

Open
#104 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I'm trying to kerchunk the medium range netcdf files from the National Water Model, which are only accessible via HTTPS from NOMADS (not S3).

I can successfully a few files and then it bombs out with a cryptic error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_1913/3841065069.py in <module>
      1 for url in urls:
----> 2     gen_json(url)

/tmp/ipykernel_1913/476567446.py in gen_json(u)
      6         h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)
      7         with fs2.open(outf, 'wb') as f:
----> 8             f.write(ujson.dumps(h5chunks.translate()).encode());

/home/conda/store/61285a2505cb2471f92bbdbcc3eccd36f27fdbd1be0fd3e864dac20b9c422482-pangeo/lib/python3.9/site-packages/kerchunk/hdf.py in translate(self)
     71         lggr.debug('Translation begins')
     72         self._transfer_attrs(self._h5f, self._zroot)
---> 73         self._h5f.visititems(self._translator)
     74         if self.inline > 0:
     75             self._do_inline(self.inline)

/home/conda/store/61285a2505cb2471f92bbdbcc3eccd36f27fdbd1be0fd3e864dac20b9c422482-pangeo/lib/python3.9/site-packages/h5py/_hl/group.py in visititems(self, func)
    610                 name = self._d(name)
    611                 return func(name, self[name])
--> 612             return h5o.visit(self.id, proxy)
    613 
    614     @with_phil

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5o.pyx in h5py.h5o.visit()

RuntimeError: Object visitation failed (wrong fractal heap header signature)

Do you think this is on the provider side or on our end?

Here's a notebook that should reproduce the issue:
https://nbviewer.org/gist/rsignell-usgs/2af84b60301a9c7c9a46aaef2b84d2fa

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked nbviewer notebook and reproduce the failure across the listed NOMADS HTTPS URLs. Trace SingleHdf5ToZarr.translate through kerchunk/hdf.py, comparing successful and failing files. Done means determining whether the failure is provider-side or in kerchunk and recording a reproducible diagnosis or clearly scoped fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.