fsspec / fsspec/kerchunk

support for sparse arrays

Open
#301 13 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I've been testing kerchunk and follow the tutorial to work with h5ad files. Unfortunately I've realized that the json reference file that I created with kerchunk will not work with xr.open_dataset, I believe this is because the original file contains sparse arrays.

Do you have any suggestions how can I use kerchunk and the reference file for this kind of data?

Below the code that can reproduce my issue:

# creating the data and saving as a h5ad file
import numpy as np
from scipy.sparse import csr_matrix
import anndata

row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 2, 2, 0, 1, 2])
data = np.array([1, 2, 3, 4, 5, 6])
data_csr = csr_matrix((data, (row, col)), shape=(3, 3))

small_data = anndata.AnnData(X=data_csr)
small_data_path = '/home/jovyan/kerchunk_tutorial/small_test.h5ad'
small_data.write_h5ad(small_data_path)


# creating the json ref file with kerchunk
from kerchunk.hdf import SingleHdf5ToZarr
import fsspec
import ujson

fs_loc = fsspec.filesystem('')
so = dict(mode='rb', anon=True, default_fill_cache=False, default_cache_type='first') # args to fs.open()
outf_test = "/home/jovyan/kerchunk_tutorial/small_test_chunk.json"
with fs_loc.open(small_data_path, **so) as infile:
    h5chunks = SingleHdf5ToZarr(infile, inline_threshold=300)
    with fs_loc.open(outf_test, 'wb') as f:
        f.write(ujson.dumps(h5chunks.translate()).encode());


# using the json ref file with xarray
import xarray as xr
ds = xr.open_dataset("reference://", engine="zarr", backend_kwargs={
                    "consolidated": False,
                    "storage_options": {"fo": outf_test}
                    })

I don't get any error, but I get an empty dataset:

<xarray.Dataset>
Dimensions:  ()
Data variables:
    *empty*

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduced Python flow using SingleHdf5ToZarr and the reference:// xarray.open_dataset call, then inspect the generated reference for the sparse h5ad data. Determine why the translated reference opens as an empty dataset and define support for exposing these sparse arrays through the reference file. Done means the supplied example opens with the expected data variables and dimensions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.