Compatibility with xarray
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 171
- Forks
- 75
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 20
Description
With the aspiration for OME-Zarr to be The One Imaging Format to Rule them All 💍 , I would like to propose compatibility with xarray. For the most part, the needs of the:
- bioimaging
- geospatial imaging
- medical imaging
- many other scientific imaging domains
overlap. A common, well-supported standard will facilitate integration and cross-pollination across communities, and avoid those I/O headaches 🤯 .
In summary, we could extend the current OME-Zarr spec to be compatible with the result of xarray.Dataset.to_zarr, in a way that adds spatial metadata, addressing #28 #12, through the xarray encoded coords using scientific imaging dimensions, x, y, z, c, t, standard in OME-Zarr, for the xarray array dimensions, making their name and order explicit #35.
Resulting consolidated metadata from idr0094
{
"metadata": {
".zattrs": {
"multiscales": [
{
"datasets": [
{
"path": "0/idr0094"
},
{
"path": "1/idr0094"
},
{
"path": "2/idr0094"
},
{
"path": "3/idr0094"
},
{
"path": "4/idr0094"
},
{
"path": "5/idr0094"
}
],
"name": "idr0094",
"version": "0.1"
}
]
},
".zgroup": {
"zarr_format": 2
},
"0/.zattrs": {},
"0/.zgroup": {
"zarr_format": 2
},
"0/c/.zarray": {
"chunks": [
3
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<u4",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
3
],
"zarr_format": 2
},
"0/c/.zattrs": {
"_ARRAY_DIMENSIONS": [
"c"
]
},
"0/idr0094/.zarray": {
"chunks": [
270,
540,
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "zstd",
"id": "blosc",
"shuffle": 1
},
"dtype": "|u1",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
1080,
1080,
3
],
"zarr_format": 2
},
"0/idr0094/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y",
"x",
"c"
],
"direction": [
[
1.0,
0.0
],
[
0.0,
1.0
]
],
"ranges": [
[
0.0,
255.0
],
[
0.0,
255.0
],
[
0.0,
255.0
]
]
},
"0/x/.zarray": {
"chunks": [
1080
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1080
],
"zarr_format": 2
},
"0/x/.zattrs": {
"_ARRAY_DIMENSIONS": [
"x"
]
},
"0/y/.zarray": {
"chunks": [
1080
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1080
],
"zarr_format": 2
},
"0/y/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y"
]
},
"1/.zattrs": {},
"1/.zgroup": {
"zarr_format": 2
},
"1/c/.zarray": {
"chunks": [
3
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<u4",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
3
],
"zarr_format": 2
},
"1/c/.zattrs": {
"_ARRAY_DIMENSIONS": [
"c"
]
},
"1/idr0094/.zarray": {
"chunks": [
64,
64,
64
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "zstd",
"id": "blosc",
"shuffle": 1
},
"dtype": "|u1",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
540,
540,
3
],
"zarr_format": 2
},
"1/idr0094/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y",
"x",
"c"
],
"direction": [
[
1.0,
0.0
],
[
0.0,
1.0
]
],
"ranges": [
[
0.0,
255.0
],
[
0.0,
255.0
],
[
0.0,
255.0
]
]
},
"1/x/.zarray": {
"chunks": [
540
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
540
],
"zarr_format": 2
},
"1/x/.zattrs": {
"_ARRAY_DIMENSIONS": [
"x"
]
},
"1/y/.zarray": {
"chunks": [
540
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
540
],
"zarr_format": 2
},
"1/y/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y"
]
},
"2/.zattrs": {},
"2/.zgroup": {
"zarr_format": 2
},
"2/c/.zarray": {
"chunks": [
3
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<u4",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
3
],
"zarr_format": 2
},
"2/c/.zattrs": {
"_ARRAY_DIMENSIONS": [
"c"
]
},
"2/idr0094/.zarray": {
"chunks": [
64,
64,
64
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "zstd",
"id": "blosc",
"shuffle": 1
},
"dtype": "|u1",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
270,
270,
3
],
"zarr_format": 2
},
"2/idr0094/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y",
"x",
"c"
],
"direction": [
[
1.0,
0.0
],
[
0.0,
1.0
]
],
"ranges": [
[
0.0,
255.0
],
[
0.0,
255.0
],
[
0.0,
255.0
]
]
},
"2/x/.zarray": {
"chunks": [
270
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
270
],
"zarr_format": 2
},
"2/x/.zattrs": {
"_ARRAY_DIMENSIONS": [
"x"
]
},
"2/y/.zarray": {
"chunks": [
270
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
270
],
"zarr_format": 2
},
"2/y/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y"
]
},
"3/.zattrs": {},
"3/.zgroup": {
"zarr_format": 2
},
"3/c/.zarray": {
"chunks": [
3
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<u4",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
3
],
"zarr_format": 2
},
"3/c/.zattrs": {
"_ARRAY_DIMENSIONS": [
"c"
]
},
"3/idr0094/.zarray": {
"chunks": [
64,
64,
64
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "zstd",
"id": "blosc",
"shuffle": 1
},
"dtype": "|u1",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
135,
135,
3
],
"zarr_format": 2
},
"3/idr0094/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y",
"x",
"c"
],
"direction": [
[
1.0,
0.0
],
[
0.0,
1.0
]
],
"ranges": [
[
0.0,
252.0
],
[
0.0,
252.0
],
[
0.0,
252.0
]
]
},
"3/x/.zarray": {
"chunks": [
135
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
135
],
"zarr_format": 2
},
"3/x/.zattrs": {
"_ARRAY_DIMENSIONS": [
"x"
]
},
"3/y/.zarray": {
"chunks": [
135
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
135
],
"zarr_format": 2
},
"3/y/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y"
]
},
"4/.zattrs": {},
"4/.zgroup": {
"zarr_format": 2
},
"4/c/.zarray": {
"chunks": [
3
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<u4",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
3
],
"zarr_format": 2
},
"4/c/.zattrs": {
"_ARRAY_DIMENSIONS": [
"c"
]
},
"4/idr0094/.zarray": {
"chunks": [
64,
64,
64
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "zstd",
"id": "blosc",
"shuffle": 1
},
"dtype": "|u1",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
67,
67,
3
],
"zarr_format": 2
},
"4/idr0094/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y",
"x",
"c"
],
"direction": [
[
1.0,
0.0
],
[
0.0,
1.0
]
],
"ranges": [
[
0.0,
182.0
],
[
0.0,
182.0
],
[
0.0,
182.0
]
]
},
"4/x/.zarray": {
"chunks": [
67
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
67
],
"zarr_format": 2
},
"4/x/.zattrs": {
"_ARRAY_DIMENSIONS": [
"x"
]
},
"4/y/.zarray": {
"chunks": [
67
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
67
],
"zarr_format": 2
},
"4/y/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y"
]
},
"5/.zattrs": {},
"5/.zgroup": {
"zarr_format": 2
},
"5/c/.zarray": {
"chunks": [
3
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<u4",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
3
],
"zarr_format": 2
},
"5/c/.zattrs": {
"_ARRAY_DIMENSIONS": [
"c"
]
},
"5/idr0094/.zarray": {
"chunks": [
64,
64,
64
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "zstd",
"id": "blosc",
"shuffle": 1
},
"dtype": "|u1",
"fill_value": null,
"filters": null,
"order": "C",
"shape": [
33,
33,
3
],
"zarr_format": 2
},
"5/idr0094/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y",
"x",
"c"
],
"direction": [
[
1.0,
0.0
],
[
0.0,
1.0
]
],
"ranges": [
[
0.0,
116.0
],
[
0.0,
116.0
],
[
0.0,
116.0
]
]
},
"5/x/.zarray": {
"chunks": [
33
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
33
],
"zarr_format": 2
},
"5/x/.zattrs": {
"_ARRAY_DIMENSIONS": [
"x"
]
},
"5/y/.zarray": {
"chunks": [
33
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f8",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
33
],
"zarr_format": 2
},
"5/y/.zattrs": {
"_ARRAY_DIMENSIONS": [
"y"
]
}
},
"zarr_consolidated_format": 1
}
Created with this script.
In this example, the array dimensions are y, x, c, i.e. not all 5 dimensions in the current standard, and in a different order. But, these differences could be removed.
After attempting a few variations on this and putting it into practice, this seems to work well.
Each scale can be used independently. Initially, I tried to avoid the use of coords and use the more concise spatial-dimension rank spacing / scale, origin / translation. However, I found that in an array-based computing environment like scientific Python, where slicing is a bread-and-butter operation, the natural validity of 1D coords that can be sliced is helpful. And, in the development of visualization tools, this is quite handy and avoids on-demand generation.
The logic for transforming the spatial metadata is here.
Additionally, there is a growing xarray community, and compatibility helps everyone. Added as an attr is a direction / orientation matrix, which is important in medical imaging.
I am interested in everyone's thoughts. I am grossly behind on GitHub notifications, but I will check in with the discussion on this issue every day or two.
CC @joshmoore @lassoan @rabernat @constantinpape @danielballan @forman
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the current OME-Zarr specification and xarray.Dataset.to_zarr, then compare the linked example metadata with xarray's encoded coordinates and dimensions. Done would require an agreed compatibility design covering spatial metadata and explicit x, y, z, c, and t dimension naming and ordering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100