support for sparse arrays
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I've been testing kerchunk and follow the tutorial to work with h5ad files. Unfortunately I've realized that the json reference file that I created with kerchunk will not work with xr.open_dataset, I believe this is because the original file contains sparse arrays.
Do you have any suggestions how can I use kerchunk and the reference file for this kind of data?
Below the code that can reproduce my issue:
# creating the data and saving as a h5ad file
import numpy as np
from scipy.sparse import csr_matrix
import anndata
row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 2, 2, 0, 1, 2])
data = np.array([1, 2, 3, 4, 5, 6])
data_csr = csr_matrix((data, (row, col)), shape=(3, 3))
small_data = anndata.AnnData(X=data_csr)
small_data_path = '/home/jovyan/kerchunk_tutorial/small_test.h5ad'
small_data.write_h5ad(small_data_path)
# creating the json ref file with kerchunk
from kerchunk.hdf import SingleHdf5ToZarr
import fsspec
import ujson
fs_loc = fsspec.filesystem('')
so = dict(mode='rb', anon=True, default_fill_cache=False, default_cache_type='first') # args to fs.open()
outf_test = "/home/jovyan/kerchunk_tutorial/small_test_chunk.json"
with fs_loc.open(small_data_path, **so) as infile:
h5chunks = SingleHdf5ToZarr(infile, inline_threshold=300)
with fs_loc.open(outf_test, 'wb') as f:
f.write(ujson.dumps(h5chunks.translate()).encode());
# using the json ref file with xarray
import xarray as xr
ds = xr.open_dataset("reference://", engine="zarr", backend_kwargs={
"consolidated": False,
"storage_options": {"fo": outf_test}
})
I don't get any error, but I get an empty dataset:
<xarray.Dataset>
Dimensions: ()
Data variables:
*empty*
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproduced Python flow using SingleHdf5ToZarr and the reference:// xarray.open_dataset call, then inspect the generated reference for the sparse h5ad data. Determine why the translated reference opens as an empty dataset and define support for exposing these sparse arrays through the reference file. Done means the supplied example opens with the expected data variables and dimensions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100