fsspec / fsspec/kerchunk

Unexpected error message when data source is unreachable

Open
#197 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I successfully created a JSON index file for a NetCDF4 dataset using SingleHdf5ToZarr. I wanted to see what would happen if I intentionally changed the templates.u URI reference in the JSON file to a non-existent URI. This would simulate a situation where the dataset is moved after the index is created.

The index file is then read using this code:

import fsspec
import xarray as xr
import matplotlib.pyplot as plt

mapper = fsspec.get_mapper('reference://', fo="output2.json")

ds = xr.open_zarr(mapper, decode_times=False)

subset = ds['hus'].isel(plev=18).isel(time=1)
print(subset)

subset.plot()
plt.show()

I was expecting some kind of "file not found" type of error on line 7 (on open_zarr) but instead I got a ValueError when calling subset.plot() on line 12. The start of the error message is: "ValueError: The input coordinate is not sorted in increasing order along axis 0.".

Line 10 prints out the subset object and shows the following:

Coordinates:
  * lat      (lat) float64 nan nan nan nan nan nan ... nan nan nan nan nan nan
  * lon      (lon) float64 nan nan nan nan nan nan ... nan nan nan nan nan nan
    plev     float64 nan
    time     float64 45.0

So it seems that instead of failing on opening the (unreachable) data file, it's trying to create the subset object but assigning nans to the coordinate values. (Curiously, it's not assigning nan to the time coordinate.)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Python snippet with the modified output2.json and trace the reference:// mapper through xr.open_zarr and the later subset access. Compare where the unreachable templates.u URI is read with when the NaN coordinates appear. Done should establish a clear, file-not-found-style failure instead of silently producing invalid coordinates, with regression coverage for this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.