SingleHdf5ToZarr error/warning with scalar utf-8/ascii dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
When I try to run SingleHdf5ToZarr on an HDF5 file with a scalar HDF5 dataset that has a variable length utf-8 string dtype or a variable length ascii bytes dtype, I get the following warning that an error was caught:
/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/kerchunk/hdf.py:497: UserWarning: The following excepion was caught and quashed while traversing HDF5
'str' object has no attribute 'extend'
Traceback (most recent call last):
File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/kerchunk/hdf.py", line 438, in _translator
za = self._zroot.create_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/hierarchy.py", line 1094, in create_dataset
return self._write_op(self._create_dataset_nosync, name, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/hierarchy.py", line 935, in _write_op
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/hierarchy.py", line 1110, in _create_dataset_nosync
a = array(data, store=self._store, path=path, chunk_store=self._chunk_store, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/creation.py", line 439, in array
z[...] = data
~^^^^^
AttributeError: 'str' object has no attribute 'extend'
warnings.warn(msg)
When I pass error="raise" to SingleHdf5ToZarr, I see the source of the error in numcodecs/json.py:
...
File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/creation.py:439, in array(data, **kwargs)
436 z = create(**kwargs)
438 # fill with data
--> 439 z[...] = data
441 # set read_only property afterwards
442 z.read_only = read_only
File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:1497, in Array.__setitem__(self, selection, value)
1495 self.set_orthogonal_selection(pure_selection, value, fields=fields)
1496 else:
-> 1497 self.set_basic_selection(pure_selection, value, fields=fields)
File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:1591, in Array.set_basic_selection(self, selection, value, fields)
1589 # handle zero-dimensional arrays
1590 if self._shape == ():
-> 1591 return self._set_basic_selection_zd(selection, value, fields=fields)
1592 else:
1593 return self._set_basic_selection_nd(selection, value, fields=fields)
File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:1974, in Array._set_basic_selection_zd(self, selection, value, fields)
1971 pass
1972 else:
1973 # encode and store
-> 1974 cdata = self._encode_chunk(chunk)
1975 self.chunk_store[ckey] = cdata
File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:2436, in Array._encode_chunk(self, chunk)
2434 if self._filters:
2435 for f in self._filters:
-> 2436 chunk = f.encode(chunk)
2438 # check object encoding
2439 if ensure_ndarray_like(chunk).dtype == object:
File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/numcodecs/json.py:59, in JSON.encode(self, buf)
57 buf = np.asarray(buf)
58 items = buf.tolist()
---> 59 items.extend((buf.dtype.str, buf.shape))
60 return self._encoder.encode(items).encode(self._text_encoding)
AttributeError: 'str' object has no attribute 'extend'
Here, buf starts as a string. items is a string. It seems like JSON.encode assumes buf is array-like.
Example code to generate the HDF5 files:
import h5py
H5_STR = h5py.string_dtype("utf-8")
H5_BYTES = h5py.string_dtype("ascii")
with h5py.File("test_str.h5", "w") as f:
f.create_dataset("data", data="test", shape=None, dtype=H5_STR)
with h5py.File("test_bytes.h5", "w") as f:
f.create_dataset("data", data=b"test", shape=None, dtype=H5_BYTES)
Example code to generate the kerchunk reference JSON:
from kerchunk.hdf import SingleHdf5ToZarr
import fsspec
import ujson
fs_read = fsspec.filesystem('') # local file system to read from
fs_write = fsspec.filesystem('') # local file system to save final jsons to
def gen_json_from_local(local_file_path, final_remote_url, outf):
with fs_read.open(local_file_path, 'rb') as infile:
h5chunks = SingleHdf5ToZarr(infile, final_remote_url, inline_threshold=300, error="raise")
with fs_write.open(outf, 'wb') as f:
f.write(ujson.dumps(h5chunks.translate()).encode())
local_file_path = "/Users/rly/Documents/NWB/kerchunk-playground/test_str.h5"
final_remote_url = "s3://..."
outf = "test_str.json" # file name to save json to
gen_json_from_local(local_file_path, final_remote_url, outf)
local_file_path = "/Users/rly/Documents/NWB/kerchunk-playground/test_bytes.h5"
final_remote_url = "s3://..."
outf = "test_bytes.json" # file name to save json to
gen_json_from_local(local_file_path, final_remote_url, outf)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in kerchunk/hdf.py at _translator, then reproduce the issue with the provided h5py scalar UTF-8 and ASCII dataset examples using SingleHdf5ToZarr with error="raise". Verify that translating both files completes without the warning or exception and that the resulting references preserve the scalar string values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100