intake / intake/intake-xarray

Subsetting before Caching

Open
#47 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
76
Forks
36
PR merge metrics
No merged PRs in 30d

Description

Question:
I am wondering if it is possible to subset a dataset (via .sel method) before the data is cached.

Reasoning:
My use case is - I would like to cache all the landsat8 data from the s3 repository for a small research (~10km * 10km) station. Currently my catalog looks like (note this subsets the landsat8 tiffs to 60 total - but eventually I would want to use the entire timeseries):

```
plugins:
source:
- module: intake_xarray
sources:
landsat8:
description: Geotiff image of Landsat8 - TESTING.
driver: rasterio
cache:
- argkey: urlpath
regex: 'landsat-pds/c1/L8/033/031/'
type: file
args:
urlpath: 's3://landsat-pds/c1/L8/033/031/LC08_L1TP_033031_{collection_date:%Y%m%d}_20170310_01_T1/LC08_L1TP_033031_{collection_date:%Y%m%d}_20170310_01_T1_B{band:d}.TIF'
chunks:
band: 1
x: 1000
y: 1000
concat_dim: band
storage_options: {'anon': True}
```
Being able to subset before caching would reduce the amount of storage significantly (see example below for my 60 tiff subset):
```
c = intake.open_catalog('test.yml')
l8 = c.landsat8
l8_r = l8.read_chunked()
l8_r.data.nbytes*1e-9
133.64736048

l8_r_sm = l8_r.sel(x=slice(minx,maxx),y=slice(miny,maxy))
l8_r_sm.nbytes*1e-9
0.19845552000000002
```
Is there currently a way of doing this? If not, how difficult would it be to add this functionality? Is it adding an extra argument to the intake-xarray driver (which would look like: _slice: {x:(xmin,xmax),y:(ymin,ymax)}_ in the catalog) or would it need to include modifications to the caching mechanisms deeper in intake?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.