dandi / dandi/dandi-cli

Remotely read/stream embargoed Zarr

Open
#1,491 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

Hi team, @aaronkanzer and I are trying to read the metadata and chunks of an embargoed Zarr (on DANDI and LINC) and are unable to. What would be the best approach to remotely access a Zarr that is apart of an embargoed Dandiset?

For the code snippet below, I can get the `zarr_path` in the `dandiarchive` S3 bucket from the File Browser of a Dandiset using the `View Asset Metadata` button, but `s3fs` also requires AWS credentials.

```python
import zarr, s3fs

access_key = 'your-access-key-id'
secret_key = 'your-secret-access-key'
session_token = 'your-session-token' # Optional, if using temporary credentials

s3 = s3fs.S3FileSystem(key=access_key, secret=secret_key, token=session_token)

bucket_name = 'dandiarchive'
zarr_path = 'path/to/your/zarr/data.zarr'

store = s3fs.S3Map(root=f'{bucket_name}/{zarr_path}', s3=s3, check=False)

zarr_array = zarr.open_array(store, mode='r')
```

For reference, I am also hitting a blocker using the DANDI API when trying to access a public or private Zarr. I am not sure if this would be related to my use case. Using the code snippet below (which is a derivative of the [OpenScope Databook streaming section](https://alleninstitute.github.io/openscope_databook/basics/stream_nwb.html)) I receive a `Response [400]`. I presume that this is because the asset is a Zarr and the response is set in [lines 143-148](https://github.com/dandi/dandi-archive/blob/6e4bf727a02985aceb0fb896736883c00929c142/dandiapi/api/views/asset.py#L143-L148). And perhaps this is related to https://github.com/dandi/dandi-cli/issues/1455.

```python
from dandi import dandiapi

dataset = "000026"
filepath = "sub-I58/ses-Hip-CT/micr/sub-I58_sample-01_chunk-01_hipCT.ome.zarr"
dandi_api_key =

client = dandiapi.DandiAPIClient(api_url="https://api.dandiarchive.org/api", token=dandi_api_key)

my_dandiset = client.get_dandiset(dandiset_id=dataset, version_id="draft")

file = my_dandiset.get_asset_by_path(filepath)

base_url = file.client.session.head(file.base_download_url)
```

Thank you.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.