fsspec / fsspec/kerchunk

SingleHdf5ToZarr error/warning with scalar utf-8/ascii dataset

Open
#387 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

When I try to run SingleHdf5ToZarr on an HDF5 file with a scalar HDF5 dataset that has a variable length utf-8 string dtype or a variable length ascii bytes dtype, I get the following warning that an error was caught:

/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/kerchunk/hdf.py:497: UserWarning: The following excepion was caught and quashed while traversing HDF5
'str' object has no attribute 'extend'
Traceback (most recent call last):
  File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/kerchunk/hdf.py", line 438, in _translator
    za = self._zroot.create_dataset(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/hierarchy.py", line 1094, in create_dataset
    return self._write_op(self._create_dataset_nosync, name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/hierarchy.py", line 935, in _write_op
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/hierarchy.py", line 1110, in _create_dataset_nosync
    a = array(data, store=self._store, path=path, chunk_store=self._chunk_store, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/creation.py", line 439, in array
    z[...] = data
    ~^^^^^
AttributeError: 'str' object has no attribute 'extend'

  warnings.warn(msg)

When I pass error="raise" to SingleHdf5ToZarr, I see the source of the error in numcodecs/json.py:

...
File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/creation.py:439, in array(data, **kwargs)
    436 z = create(**kwargs)
    438 # fill with data
--> 439 z[...] = data
    441 # set read_only property afterwards
    442 z.read_only = read_only

File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:1497, in Array.__setitem__(self, selection, value)
   1495     self.set_orthogonal_selection(pure_selection, value, fields=fields)
   1496 else:
-> 1497     self.set_basic_selection(pure_selection, value, fields=fields)

File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:1591, in Array.set_basic_selection(self, selection, value, fields)
   1589 # handle zero-dimensional arrays
   1590 if self._shape == ():
-> 1591     return self._set_basic_selection_zd(selection, value, fields=fields)
   1592 else:
   1593     return self._set_basic_selection_nd(selection, value, fields=fields)

File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:1974, in Array._set_basic_selection_zd(self, selection, value, fields)
   1971         pass
   1972 else:
   1973     # encode and store
-> 1974     cdata = self._encode_chunk(chunk)
   1975     self.chunk_store[ckey] = cdata

File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/zarr/core.py:2436, in Array._encode_chunk(self, chunk)
   2434 if self._filters:
   2435     for f in self._filters:
-> 2436         chunk = f.encode(chunk)
   2438 # check object encoding
   2439 if ensure_ndarray_like(chunk).dtype == object:

File ~/mambaforge/envs/kerchunk/lib/python3.11/site-packages/numcodecs/json.py:59, in JSON.encode(self, buf)
     57 buf = np.asarray(buf)
     58 items = buf.tolist()
---> 59 items.extend((buf.dtype.str, buf.shape))
     60 return self._encoder.encode(items).encode(self._text_encoding)

AttributeError: 'str' object has no attribute 'extend'

Here, buf starts as a string. items is a string. It seems like JSON.encode assumes buf is array-like.

Example code to generate the HDF5 files:

import h5py

H5_STR = h5py.string_dtype("utf-8")
H5_BYTES = h5py.string_dtype("ascii")

with h5py.File("test_str.h5", "w") as f:
    f.create_dataset("data", data="test", shape=None, dtype=H5_STR)

with h5py.File("test_bytes.h5", "w") as f:
    f.create_dataset("data", data=b"test", shape=None, dtype=H5_BYTES)

Example code to generate the kerchunk reference JSON:

from kerchunk.hdf import SingleHdf5ToZarr
import fsspec
import ujson

fs_read = fsspec.filesystem('')  # local file system to read from
fs_write = fsspec.filesystem('')  # local file system to save final jsons to

def gen_json_from_local(local_file_path, final_remote_url, outf):
    with fs_read.open(local_file_path, 'rb') as infile:
        h5chunks = SingleHdf5ToZarr(infile, final_remote_url, inline_threshold=300, error="raise")

        with fs_write.open(outf, 'wb') as f:
            f.write(ujson.dumps(h5chunks.translate()).encode())

local_file_path = "/Users/rly/Documents/NWB/kerchunk-playground/test_str.h5"
final_remote_url = "s3://..."
outf = "test_str.json"  # file name to save json to
gen_json_from_local(local_file_path, final_remote_url, outf)

local_file_path = "/Users/rly/Documents/NWB/kerchunk-playground/test_bytes.h5"
final_remote_url = "s3://..."
outf = "test_bytes.json"  # file name to save json to
gen_json_from_local(local_file_path, final_remote_url, outf)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in kerchunk/hdf.py at _translator, then reproduce the issue with the provided h5py scalar UTF-8 and ASCII dataset examples using SingleHdf5ToZarr with error="raise". Verify that translating both files completes without the warning or exception and that the resulting references preserve the scalar string values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.