Issue kerchunking NWM medium range netcdf via HTTPS
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm trying to kerchunk the medium range netcdf files from the National Water Model, which are only accessible via HTTPS from NOMADS (not S3).
I can successfully a few files and then it bombs out with a cryptic error:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
/tmp/ipykernel_1913/3841065069.py in <module>
1 for url in urls:
----> 2 gen_json(url)
/tmp/ipykernel_1913/476567446.py in gen_json(u)
6 h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)
7 with fs2.open(outf, 'wb') as f:
----> 8 f.write(ujson.dumps(h5chunks.translate()).encode());
/home/conda/store/61285a2505cb2471f92bbdbcc3eccd36f27fdbd1be0fd3e864dac20b9c422482-pangeo/lib/python3.9/site-packages/kerchunk/hdf.py in translate(self)
71 lggr.debug('Translation begins')
72 self._transfer_attrs(self._h5f, self._zroot)
---> 73 self._h5f.visititems(self._translator)
74 if self.inline > 0:
75 self._do_inline(self.inline)
/home/conda/store/61285a2505cb2471f92bbdbcc3eccd36f27fdbd1be0fd3e864dac20b9c422482-pangeo/lib/python3.9/site-packages/h5py/_hl/group.py in visititems(self, func)
610 name = self._d(name)
611 return func(name, self[name])
--> 612 return h5o.visit(self.id, proxy)
613
614 @with_phil
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/h5o.pyx in h5py.h5o.visit()
RuntimeError: Object visitation failed (wrong fractal heap header signature)
Do you think this is on the provider side or on our end?
Here's a notebook that should reproduce the issue:
https://nbviewer.org/gist/rsignell-usgs/2af84b60301a9c7c9a46aaef2b84d2fa
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked nbviewer notebook and reproduce the failure across the listed NOMADS HTTPS URLs. Trace SingleHdf5ToZarr.translate through kerchunk/hdf.py, comparing successful and failing files. Done means determining whether the failure is provider-side or in kerchunk and recording a reproducible diagnosis or clearly scoped fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100