scverse / scverse/spatialdata

Jupyter kernel crashing with sdata.write

Open
#827 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
394
Forks
95
Avg merge
4d 3h
Merged PRs (30d)
7

Description

Describe the bug
I am writing large sdata objects to zarr, and the Kernel fails in an unpredictable manner.

I parse the image into sdata, a large mIF image (15, 44470, 73167) (8bit), with scale factors (5,5) to create a multiscale object. Then writing that simple sdata object seems to fail, (it takes about 20min, so only tried twice).

Before I send over this large data, are there any expected limitations from writing sdata objects into Zarr in a Jupyter notebook?
My naive concerns think about:

  1. chunking
    (what if chunk size is larger than downscaled image size?)
    (can I chunk different scales dynamically? I use the parser to chunk.)
  2. Hardware
    (I use M2 Macbook Pro).

This kind of kernel failures are particularly frustrating because they corrupt the zarr object, I was writing some new elements (another very large image) and it crashed, and it killed the object.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure in a Jupyter notebook with the reported large multiscale sdata object, image dimensions, and scale factors. Investigate chunking and hardware limits during sdata.write to Zarr; done means the crash cause or supported limitations are identified and the risk of corrupting the Zarr object is addressed or documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.