missing object_codec for object array
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I am trying to create references for this netCDF file which is a collection of ~30 xArray dataSets. Each dataSet has 8 HDF5 datasets. This file was written with xArray.
I make it through the root group and the first dataset then I get this error on the second dataset in the group:
Exception has occurred: ValueError
missing object_codec for object array
File "/Users/metadatagamechanger/GitRepositories/fsspec-reference-maker/fsspec_reference_maker/hdf.py", line 177, in _translator
za = self._zroot.create_dataset(h5obj.name, shape=h5obj.shape,
File "/Users/metadatagamechanger/GitRepositories/fsspec-reference-maker/fsspec_reference_maker/hdf.py", line 71, in translate
self._h5f.visititems(self._translator)
File "/Users/metadatagamechanger/Library/Mobile Documents/comappleCloudDocs/Documents/MetadataGameChanger/ProjectsAndPlans/UNAVCO/CommonContainer/referenceFileSystem/makeReferences.py", line 10, in
h5t = h5chunks.translate()
On line: za = self._zroot.create_dataset(h5obj.name, shape=h5obj.shape,
dtype=h5obj.dtype,
chunks=h5obj.chunks or False,
fill_value=h5obj.fillvalue,
compression=compression,
filters=filters,
overwrite=True)
The json so far ends with the reference to the data in the first dataset. The h5ls for the dataset the error comes from is:
/AMC2/Solution Dataset {7025/7025}
Attribute: DIMENSION_LIST {1}
Type: variable length of
object reference
Data: (DATASET-1:529616)
Attribute: _Netcdf4Dimid scalar
Type: native int
Data: 2
Location: 1:932878
Links: 1
Storage: 56200 logical bytes, 112400 allocated bytes, 50.00% utilization
Type: variable-length null-terminated UTF-8 string
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec_reference_maker/hdf.py, especially translate and _translator around the create_dataset call at line 177. Reproduce the failure with the linked netCDF file and inspect the second dataset's dtype and filters, focusing on the object array and variable-length string metadata shown by h5ls. Done means the file translates past that dataset and produces references without the missing object_codec error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100