Remotely read/stream embargoed Zarr
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
Hi team, @aaronkanzer and I are trying to read the metadata and chunks of an embargoed Zarr (on DANDI and LINC) and are unable to. What would be the best approach to remotely access a Zarr that is apart of an embargoed Dandiset?
For the code snippet below, I can get the `zarr_path` in the `dandiarchive` S3 bucket from the File Browser of a Dandiset using the `View Asset Metadata` button, but `s3fs` also requires AWS credentials.
```python
import zarr, s3fs
access_key = 'your-access-key-id'
secret_key = 'your-secret-access-key'
session_token = 'your-session-token' # Optional, if using temporary credentials
s3 = s3fs.S3FileSystem(key=access_key, secret=secret_key, token=session_token)
bucket_name = 'dandiarchive'
zarr_path = 'path/to/your/zarr/data.zarr'
store = s3fs.S3Map(root=f'{bucket_name}/{zarr_path}', s3=s3, check=False)
zarr_array = zarr.open_array(store, mode='r')
```
For reference, I am also hitting a blocker using the DANDI API when trying to access a public or private Zarr. I am not sure if this would be related to my use case. Using the code snippet below (which is a derivative of the [OpenScope Databook streaming section](https://alleninstitute.github.io/openscope_databook/basics/stream_nwb.html)) I receive a `Response [400]`. I presume that this is because the asset is a Zarr and the response is set in [lines 143-148](https://github.com/dandi/dandi-archive/blob/6e4bf727a02985aceb0fb896736883c00929c142/dandiapi/api/views/asset.py#L143-L148). And perhaps this is related to https://github.com/dandi/dandi-cli/issues/1455.
```python
from dandi import dandiapi
dataset = "000026"
filepath = "sub-I58/ses-Hip-CT/micr/sub-I58_sample-01_chunk-01_hipCT.ome.zarr"
dandi_api_key =
client = dandiapi.DandiAPIClient(api_url="https://api.dandiarchive.org/api", token=dandi_api_key)
my_dandiset = client.get_dandiset(dandiset_id=dataset, version_id="draft")
file = my_dandiset.get_asset_by_path(filepath)
base_url = file.client.session.head(file.base_download_url)
```
Thank you.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.