Subsetting before Caching
- Dominant language
- Python
- Stars
- 76
- Forks
- 36
- PR merge metrics
- No merged PRs in 30d
Description
Question:
I am wondering if it is possible to subset a dataset (via .sel method) before the data is cached.
Reasoning:
My use case is - I would like to cache all the landsat8 data from the s3 repository for a small research (~10km * 10km) station. Currently my catalog looks like (note this subsets the landsat8 tiffs to 60 total - but eventually I would want to use the entire timeseries):
```
plugins:
source:
- module: intake_xarray
sources:
landsat8:
description: Geotiff image of Landsat8 - TESTING.
driver: rasterio
cache:
- argkey: urlpath
regex: 'landsat-pds/c1/L8/033/031/'
type: file
args:
urlpath: 's3://landsat-pds/c1/L8/033/031/LC08_L1TP_033031_{collection_date:%Y%m%d}_20170310_01_T1/LC08_L1TP_033031_{collection_date:%Y%m%d}_20170310_01_T1_B{band:d}.TIF'
chunks:
band: 1
x: 1000
y: 1000
concat_dim: band
storage_options: {'anon': True}
```
Being able to subset before caching would reduce the amount of storage significantly (see example below for my 60 tiff subset):
```
c = intake.open_catalog('test.yml')
l8 = c.landsat8
l8_r = l8.read_chunked()
l8_r.data.nbytes*1e-9
133.64736048
l8_r_sm = l8_r.sel(x=slice(minx,maxx),y=slice(miny,maxy))
l8_r_sm.nbytes*1e-9
0.19845552000000002
```
Is there currently a way of doing this? If not, how difficult would it be to add this functionality? Is it adding an extra argument to the intake-xarray driver (which would look like: _slice: {x:(xmin,xmax),y:(ymin,ymax)}_ in the catalog) or would it need to include modifications to the caching mechanisms deeper in intake?
Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.