fsspec / fsspec/kerchunk

missing object_codec for object array

Open
#32 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I am trying to create references for this netCDF file which is a collection of ~30 xArray dataSets. Each dataSet has 8 HDF5 datasets. This file was written with xArray.

I make it through the root group and the first dataset then I get this error on the second dataset in the group:
Exception has occurred: ValueError
missing object_codec for object array
File "/Users/metadatagamechanger/GitRepositories/fsspec-reference-maker/fsspec_reference_maker/hdf.py", line 177, in _translator
za = self._zroot.create_dataset(h5obj.name, shape=h5obj.shape,
File "/Users/metadatagamechanger/GitRepositories/fsspec-reference-maker/fsspec_reference_maker/hdf.py", line 71, in translate
self._h5f.visititems(self._translator)
File "/Users/metadatagamechanger/Library/Mobile Documents/comappleCloudDocs/Documents/MetadataGameChanger/ProjectsAndPlans/UNAVCO/CommonContainer/referenceFileSystem/makeReferences.py", line 10, in
h5t = h5chunks.translate()

On line: za = self._zroot.create_dataset(h5obj.name, shape=h5obj.shape,
dtype=h5obj.dtype,
chunks=h5obj.chunks or False,
fill_value=h5obj.fillvalue,
compression=compression,
filters=filters,
overwrite=True)

The json so far ends with the reference to the data in the first dataset. The h5ls for the dataset the error comes from is:
/AMC2/Solution Dataset {7025/7025}
Attribute: DIMENSION_LIST {1}
Type: variable length of
object reference
Data: (DATASET-1:529616)
Attribute: _Netcdf4Dimid scalar
Type: native int
Data: 2
Location: 1:932878
Links: 1
Storage: 56200 logical bytes, 112400 allocated bytes, 50.00% utilization
Type: variable-length null-terminated UTF-8 string

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec_reference_maker/hdf.py, especially translate and _translator around the create_dataset call at line 177. Reproduce the failure with the linked netCDF file and inspect the second dataset's dtype and filters, focusing on the object array and variable-length string metadata shown by h5ls. Done means the file translates past that dataset and produces references without the missing object_codec error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.