fsspec / fsspec/kerchunk

TypeError: Invalid value for attr '_FillValue': b''.

Open
#121 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I'm trying to combine three netcdf file jsons using the following notebook:
https://nbviewer.org/gist/6117f63ba1b266ea713249b98492ce76
(which could be reproduced if you have AWS credentials -- it reads from a requester pays bucket)

I'm getting this error, which I really don't understand, since we are not creating NetCDF files and the original NetCDF files don't have any fill values.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File <timed exec>:2, in <module>

File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/kerchunk/combine.py:70, in MultiZarrToZarr.translate(self, outpath, template_count)
     55 """
     56 Translate the combined reference files and write to new file
     57 
   (...)
     65 
     66 """
     69 ds, ds0, fss = self._determine_dims()
---> 70 out = self._build_output(ds, ds0, fss)
     71 self.output = self._consolidate(out, template_count=template_count)
     73 if outpath:

File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/kerchunk/combine.py:171, in MultiZarrToZarr._build_output(self, ds, ds0, fss)
    169 out = {}
    170 logger.debug("write zarr metadata")
--> 171 ds.to_zarr(out, chunk_store={}, compute=False,
    172            consolidated=False)  # fills in metadata&coords
    173 z = zarr.open_group(out, mode='a')
    174 accum_dim = list(self.concat_dims.union(self.extra_dims))[0]  # only ever one dim for now

File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/core/dataset.py:2035, in Dataset.to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
   2032 if encoding is None:
   2033     encoding = {}
-> 2035 return to_zarr(
   2036     self,
   2037     store=store,
   2038     chunk_store=chunk_store,
   2039     storage_options=storage_options,
   2040     mode=mode,
   2041     synchronizer=synchronizer,
   2042     group=group,
   2043     encoding=encoding,
   2044     compute=compute,
   2045     consolidated=consolidated,
   2046     append_dim=append_dim,
   2047     region=region,
   2048     safe_chunks=safe_chunks,
   2049 )

File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/backends/api.py:1375, in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
   1373 # validate Dataset keys, DataArray names, and attr keys/values
   1374 _validate_dataset_names(dataset)
-> 1375 _validate_attrs(dataset)
   1377 if region is not None:
   1378     _validate_region(dataset, region)

File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/backends/api.py:175, in _validate_attrs(dataset, invalid_netcdf)
    173 for variable in dataset.variables.values():
    174     for k, v in variable.attrs.items():
--> 175         check_attr(k, v, valid_types)

File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/backends/api.py:162, in _validate_attrs.<locals>.check_attr(name, value, valid_types)
    156     raise TypeError(
    157         f"Invalid name for attr: {name!r} must be a string for "
    158         "serialization to netCDF files"
    159     )
    161 if not isinstance(value, valid_types):
--> 162     raise TypeError(
    163         f"Invalid value for attr {name!r}: {value!r}. For serialization to "
    164         "netCDF files, its value must be of one of the following types: "
    165         f"{', '.join([vtype.__name__ for vtype in valid_types])}"
    166     )

TypeError: Invalid value for attr '_FillValue': b''. For serialization to netCDF files, its value must be of one of the following types: str, Number, ndarray, number, list, tuple

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the linked notebook and inspect MultiZarrToZarr.translate and _build_output in kerchunk/combine.py, where the traceback reaches xarray Dataset.to_zarr. Check how the combined dataset metadata reaches xarray's attribute validation; done means the three reference JSONs combine without the reported TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.