intake / intake/intake-stac

add s3 URL support to intake-stac

Open
#48 6 comments 1 reaction 0 assignees View on GitHub
documentation enhancement
Dominant language
Python
Stars
113
Forks
25
PR merge metrics
No merged PRs in 30d

Description

I was recently part of a group trying to use intake-stac to bring some files into dask from s3. Unfortunately, the data in question was not public and neither were the catalog files. So I wanted to use s3-style URLs for everything. Unfortunately, when I tried the following:

```
import intake
from intake import open_stac_catalog
cat = open_stac_catalog('s3://put-real-bucket-here/catalog.json')
```

I got the error:

```
---------------------------------------------------------------------------
STACError Traceback (most recent call last)
in
----> 1 cat = open_stac_catalog('s3://sit-giovanni-website/catalog.json')

~/Software/python3/envs/stac/lib/python3.8/site-packages/intake_stac/catalog.py in __init__(self, stac_obj, **kwargs)
30 self._stac_obj = stac_obj
31 elif isinstance(stac_obj, str):
---> 32 self._stac_obj = self._stac_cls.open(stac_obj)
33 else:
34 raise ValueError(

~/Software/python3/envs/stac/lib/python3.8/site-packages/satstac/thing.py in open(cls, filename)
56 dat = json.loads(dat)
57 else:
---> 58 raise STACError('%s does not exist locally' % filename)
59 return cls(dat, filename=filename)
60

STACError: s3://put-real-bucket-here/catalog.json does not exist locally
```

It looks to me as though STAC thinks this is a file path rather than an S3 URL. Our time was short and I couldn't figure out if there was some other way to get STAC to take an S3 URL.

At the same time, we were hoping to put s3 URLs in our item catalog entries. E.g. -

```
{
"id": "AIRX3STD_006_SurfAirTemp_A/AIRX3STD_006_SurfAirTemp_A_2002_08.parquet",
"stac_version": "0.9.0",
"title": "Air temperature at surface (Daytime/Ascending) [AIRS AIRX3STD v006] for 2002-08-01 00:00:00+00:00 - 2002-08-31 23:59:59+00:00",
"description": "Parquet file containing data from Air temperature at surface (Daytime/Ascending) [AIRS AIRX3STD v006] for 2002-08-01 00:00:00+00:00 - 2002-08-31 23:59:59+00:00",
"type": "Feature",
"bbox": [
-180.0,
-90.0,
180.0,
90.0
],
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
-180.0,
-90.0
],
[
-180.0,
90.0
],
[
180.0,
90.0
],
[
180.0,
-90.0
],
[
-180.0,
-90.0
]
]
]
},
"properties": {
"datetime": "2002-08-01T00:00:00Z",
"start_datetime": "2002-08-01T00:00:00Z",
"end_datetime": "2002-08-31T23:59:59Z",
"created": "2020-07-27T18:35:54Z",
"license": "Apache",
"platform": "AIRS",
"instruments": [
"AIRS"
],
"mission": "AIRS"
},
"assets": {
"AIRX3STD_006_SurfAirTemp_A_2002_08.parquet": {
"href": "s3://fill-in-real-bucket-here/some/path/AIRX3STD_006_SurfAirTemp_A_2002_08.parquet",
"title": "Air temperature at surface (Daytime/Ascending) [AIRS AIRX3STD v006], 2002-08-01 00:00:00+00:00 - 2002-08-31 23:59:59+00:00 (Parquet)",
"type": "parquet",
"roles": [
"data"
]
}
},
"links": [
{
"rel": "self",
"href": "N/A"
}
]
}
```

This one may be more of a stretch. I don't know if the STAC spec has s3-style URLs in mind. My two minute evaluation of the item spec (https://github.com/radiantearth/stac-spec/blob/master/item-spec/json-schema/item.json) is inconclusive.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.