TypeError: Invalid value for attr '_FillValue': b''.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm trying to combine three netcdf file jsons using the following notebook:
https://nbviewer.org/gist/6117f63ba1b266ea713249b98492ce76
(which could be reproduced if you have AWS credentials -- it reads from a requester pays bucket)
I'm getting this error, which I really don't understand, since we are not creating NetCDF files and the original NetCDF files don't have any fill values.
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
File <timed exec>:2, in <module>
File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/kerchunk/combine.py:70, in MultiZarrToZarr.translate(self, outpath, template_count)
55 """
56 Translate the combined reference files and write to new file
57
(...)
65
66 """
69 ds, ds0, fss = self._determine_dims()
---> 70 out = self._build_output(ds, ds0, fss)
71 self.output = self._consolidate(out, template_count=template_count)
73 if outpath:
File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/kerchunk/combine.py:171, in MultiZarrToZarr._build_output(self, ds, ds0, fss)
169 out = {}
170 logger.debug("write zarr metadata")
--> 171 ds.to_zarr(out, chunk_store={}, compute=False,
172 consolidated=False) # fills in metadata&coords
173 z = zarr.open_group(out, mode='a')
174 accum_dim = list(self.concat_dims.union(self.extra_dims))[0] # only ever one dim for now
File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/core/dataset.py:2035, in Dataset.to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
2032 if encoding is None:
2033 encoding = {}
-> 2035 return to_zarr(
2036 self,
2037 store=store,
2038 chunk_store=chunk_store,
2039 storage_options=storage_options,
2040 mode=mode,
2041 synchronizer=synchronizer,
2042 group=group,
2043 encoding=encoding,
2044 compute=compute,
2045 consolidated=consolidated,
2046 append_dim=append_dim,
2047 region=region,
2048 safe_chunks=safe_chunks,
2049 )
File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/backends/api.py:1375, in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
1373 # validate Dataset keys, DataArray names, and attr keys/values
1374 _validate_dataset_names(dataset)
-> 1375 _validate_attrs(dataset)
1377 if region is not None:
1378 _validate_region(dataset, region)
File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/backends/api.py:175, in _validate_attrs(dataset, invalid_netcdf)
173 for variable in dataset.variables.values():
174 for k, v in variable.attrs.items():
--> 175 check_attr(k, v, valid_types)
File /home/conda/store/26851070ae96f50ac395255946c7876bb86f109ca9ae22af854d21e68393d114-pangeo/lib/python3.9/site-packages/xarray/backends/api.py:162, in _validate_attrs.<locals>.check_attr(name, value, valid_types)
156 raise TypeError(
157 f"Invalid name for attr: {name!r} must be a string for "
158 "serialization to netCDF files"
159 )
161 if not isinstance(value, valid_types):
--> 162 raise TypeError(
163 f"Invalid value for attr {name!r}: {value!r}. For serialization to "
164 "netCDF files, its value must be of one of the following types: "
165 f"{', '.join([vtype.__name__ for vtype in valid_types])}"
166 )
TypeError: Invalid value for attr '_FillValue': b''. For serialization to netCDF files, its value must be of one of the following types: str, Number, ndarray, number, list, tuple
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the linked notebook and inspect MultiZarrToZarr.translate and _build_output in kerchunk/combine.py, where the traceback reaches xarray Dataset.to_zarr. Check how the combined dataset metadata reaches xarray's attribute validation; done means the three reference JSONs combine without the reported TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100