fsspec / fsspec/kerchunk

Issue using `tiff_to_zarr`

Open
#313 14 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

Hi there @martindurant,

I've started playing around with Kerchunk's tiff_to_zarr functionality and ran into an issue when trying to open up the reference file. In short, the tiff_to_zarr works successfully, but I get the error: ContainsArrayError: path '' contains an array. Hopefully this is a simple user error!
I've successfully used tiff_to_zarr on the kerchunk/tests/lcmap_tiny_cog_2019.tif and this Arctic DEM without incident.

Thanks in advance!

I've included a code snippet below and a example of the generated reference file.

Generation script:



import fsspec
import xarray as xr
import ujson
import imagecodecs.numcodecs
imagecodecs.numcodecs.register_codecs()

from kerchunk.tiff import tiff_to_zarr

hansen_url = 'https://storage.googleapis.com/earthenginepartners-hansen/GFC-2021-v1.9/Hansen_GFC-2021-v1.9_gain_50N_130W.tif'
ex_path = 'https://github.com/fsspec/kerchunk/blob/main/kerchunk/tests/lcmap_tiny_cog_2019.tif?raw=true'

hansen_ds = xr.open_dataset(hansen_url,engine='rasterio')


ex_ds = xr.open_dataset(ex_path,engine='rasterio')



zt = tiff_to_zarr(hansen_url)
with open("hansen.json", "wb") as f:
    f.write(ujson.dumps(zt).encode())

m = fsspec.get_mapper("reference://", fo='hansen.json')
hansen_kerchunk_ds = xr.open_dataset(
    m, engine="zarr", backend_kwargs={"consolidated": False}
)

examples of refs for both datasets:

Hansen:

{
	".zattrs": "{\"_ARRAY_DIMENSIONS\":[\"Y\",\"X\"],\"KeyDirectoryVersion\":1,\"KeyRevision\":1,\"KeyRevisionMinor\":0,\"GTModelTypeGeoKey\":2,\"GTRasterTypeGeoKey\":1,\"GeographicTypeGeoKey\":4326,\"GeogCitationGeoKey\":\"WGS 84\",\"GeogAngularUnitsGeoKey\":9102,\"GeogSemiMajorAxisGeoKey\":6378137.0,\"GeogInvFlatteningGeoKey\":298.257223563,\"ModelPixelScale\":[0.00025,0.00025,0.0],\"ModelTiepoint\":[0.0,0.0,0.0,-130.0,50.0,0.0]}",
	".zarray": "{\n \"chunks\": [\n  1,\n  40000\n ],\n \"compressor\": {\n  \"id\": \"imagecodecs_lzw\"\n },\n \"dtype\": \"|u1\",\n \"fill_value\": 0,\n \"filters\": null,\n \"order\": \"C\",\n \"shape\": [\n  40000,\n  40000\n ],\n \"zarr_format\": 2\n}",
	"0.0": ["https:\/\/storage.googleapis.com\/earthenginepartners-hansen\/GFC-2021-v1.9\/Hansen_GFC-2021-v1.9_gain_50N_130W.tif", 320378, 777],
	"1.0": ["https:\/\/storage.googleapis.com\/earthenginepartners-hansen\/GFC-2021-v1.9\/Hansen_GFC-2021-v1.9_gain_50N_130W.tif", 321155, 796], ...}

Kerchunk test tiff:


{
	".zgroup": "{\n \"zarr_format\": 2\n}",
	".zattrs": "{\"multiscales\":[{\"datasets\":[{\"path\":\"0\"},{\"path\":\"1\"},{\"path\":\"2\"}],\"metadata\":{},\"name\":\"\",\"version\":\"0.1\"}],\"OVR_RESAMPLING_ALG\":\"NEAREST\",\"LAYOUT\":\"IFDS_BEFORE_DATA\",\"BLOCK_ORDER\":\"ROW_MAJOR\",\"BLOCK_LEADER\":\"SIZE_AS_UINT4\",\"BLOCK_TRAILER\":\"LAST_4_BYTES_REPEATED\",\"KNOWN_INCOMPATIBLE_EDITION\":\"NO\",\"KeyDirectoryVersion\":1,\"KeyRevision\":1,\"KeyRevisionMinor\":0,\"GTModelTypeGeoKey\":1,\"GTRasterTypeGeoKey\":1,\"GTCitationGeoKey\":\"Albers\",\"GeographicTypeGeoKey\":4326,\"GeogCitationGeoKey\":\"WGS 84\",\"GeogAngularUnitsGeoKey\":9102,\"GeogSemiMajorAxisGeoKey\":6378140.0,\"GeogInvFlatteningGeoKey\":298.256999999996,\"ProjectedCSTypeGeoKey\":32767,\"ProjectionGeoKey\":32767,\"ProjCoordTransGeoKey\":11,\"ProjLinearUnitsGeoKey\":9001,\"ProjStdParallel1GeoKey\":29.5,\"ProjStdParallel2GeoKey\":45.5,\"ProjNatOriginLongGeoKey\":-96.0,\"ProjNatOriginLatGeoKey\":23.0,\"ProjFalseEastingGeoKey\":0.0,\"ProjFalseNorthingGeoKey\":0.0,\"ModelPixelScale\":[30.0,30.0,0.0],\"ModelTiepoint\":[0.0,0.0,0.0,-1801185.0,2700405.0,0.0]}",
	"0\/.zattrs": "{\n \"_ARRAY_DIMENSIONS\": [\n  \"Y\",\n  \"X\"\n ]\n}",
	"0\/.zarray": "{\n \"chunks\": [\n  512,\n  512\n ],\n \"compressor\": {\n  \"id\": \"zlib\"\n },\n \"dtype\": \"|u1\",\n \"fill_value\": 0,\n \"filters\": null,\n \"order\": \"C\",\n \"shape\": [\n  2048,\n  2048\n ],\n \"zarr_format\": 2\n}",
	"1\/.zattrs": "{\n \"_ARRAY_DIMENSIONS\": [\n  \"Y1\",\n  \"X1\"\n ]\n}",
	"1\/.zarray": "{\n \"chunks\": [\n  128,\n  128\n ],\n \"compressor\": {\n  \"id\": \"zlib\"\n },\n \"dtype\": \"|u1\",\n \"fill_value\": 0,\n \"filters\": null,\n \"order\": \"C\",\n \"shape\": [\n  1024,\n  1024\n ],\n \"zarr_format\": 2\n}",
	"2\/.zattrs": "{\n \"_ARRAY_DIMENSIONS\": [\n  \"Y2\",\n  \"X2\"\n ]\n}",
	"2\/.zarray": "{\n \"chunks\": [\n  128,\n  128\n ],\n \"compressor\": {\n  \"id\": \"zlib\"\n },\n \"dtype\": \"|u1\",\n \"fill_value\": 0,\n \"filters\": null,\n \"order\": \"C\",\n \"shape\": [\n  512,\n  512\n ],\n \"zarr_format\": 2\n}",
	"0\/0.0": ["https:\/\/github.com\/fsspec\/kerchunk\/blob\/main\/kerchunk\/tests\/lcmap_tiny_cog_2019.tif?raw=true", 114079, 13584],
	"0\/0.1": ["https:\/\/github.com\/fsspec\/kerchunk\/blob\/main\/kerchunk\/tests\/lcmap_tiny_cog_2019.tif?raw=true", 127671, 19626], ...}

repr's of both ref datasets:

image image Traceback:
---------------------------------------------------------------------------
ContainsArrayError                        Traceback (most recent call last)
Cell In[49], line 2
      1 m = fsspec.get_mapper("reference:[//](https://github.com/fsspec/kerchunk/issues/313)", fo='hansen.json')
----> 2 hansen_kerchunk_ds = xr.open_dataset(
      3     m, engine="zarr", backend_kwargs={"consolidated": False}
      4 )

File [~/opt/anaconda3/envs/install/envs/test_env/lib/python3.9/site-packages/xarray/backends/api.py:526](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/global_forest_watch/~/opt/anaconda3/envs/install/envs/test_env/lib/python3.9/site-packages/xarray/backends/api.py:526), in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, backend_kwargs, **kwargs)
    514 decoders = _resolve_decoders_kwargs(
    515     decode_cf,
    516     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)
    522     decode_coords=decode_coords,
    523 )
    525 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 526 backend_ds = backend.open_dataset(
    527     filename_or_obj,
    528     drop_variables=drop_variables,
    529     **decoders,
    530     **kwargs,
    531 )
    532 ds = _dataset_from_backend_dataset(
    533     backend_ds,
    534     filename_or_obj,
...
-> 1442             raise ContainsArrayError(path)
   1443         raise GroupNotFoundError(path)
   1445 elif mode == 'w':

ContainsArrayError: path '' contains an array

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the kerchunk.tiff.tiff_to_zarr entry point and reproduce the xarray.open_dataset call using the generated hansen.json reference. Compare its structure with the working lcmap_tiny_cog_2019.tif reference and trace the source of ContainsArrayError. Done means the reported reference opens successfully or the unsupported TIFF structure and expected behavior are clearly diagnosed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.