pytroll / pytroll/pyresample

Status of dask support in pyresample?

Open
#206 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
385
Forks
102
Avg merge
4d 2h
Merged PRs (30d)
9

Description

The latest pyresample docs state:

Interfaces to XArray objects (including dask array support) are provided in separate Resampler class interfaces and are in active development.

However, there is no further mention of xarray / dask support in the rest of the docs. There seem to be a few dask issues (e.g. #148), but I could not ascertain the status of xarray / dask support based on browsing the docs and the repo.

Could you clarify where things stand? My use case is that I would like to use pyresample lazily on dask arrays, where the data is chunked contiguously in space but has many samples in time. Here's what I have tried, representing each timestep as a different channel:

import numpy as np
import dask.array as da
from pyresample import image, geometry

area_def = geometry.AreaDefinition('areaD', 'Europe (3km, HRV, VTC)', 'areaD',
                               {'a': '6378144.0', 'b': '6356759.0',
                                'lat_0': '50.00', 'lat_ts': '50.00',
                                'lon_0': '8.00', 'proj': 'stere'},
                               800, 800,
                               [-1370912.72, -909968.64,
                                1029087.28, 1490031.36])
msg_area = geometry.AreaDefinition('msg_full', 'Full globe MSG image 0 degrees',
                               'msg_full',
                               {'a': '6378169.0', 'b': '6356584.0',
                                'h': '35785831.0', 'lon_0': '0',
                                'proj': 'geos'},
                               3712, 3712,
                               [-5568742.4, -5568742.4,
                                5568742.4, 5568742.4])

# Here I have 10 "timesteps" (pyresample calls these "channels")
# The channels are the final axis (axis=2) of the array
# works if I use numpy
# data = np.random.rand(3712, 3712, 10)
data = da.random.random((3712, 3712, 10), chunks=(3712, 3712, 1))

msg_con_nn = image.ImageContainerNearest(data, msg_area, radius_of_influence=50000)
msg_con_nn.resample(area_def)

I would expect this to lazily return a dask array with the same chunk structure as the input array, with resampling performed on demand as the chunks are loaded. Instead I hit an error when creating the ImageContainerNearest object:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-d3a77de18e5c> in <module>
      1 #data = np.random.rand(3712, 3712, 10)
      2 data = da.random.random((3712, 3712, 10), chunks=(3712, 3712, 1))
----> 3 msg_con_nn = image.ImageContainerNearest(data, msg_area, radius_of_influence=50000)

/srv/conda/envs/notebook/lib/python3.7/site-packages/pyresample/image.py in __init__(self, image_data, geo_def, radius_of_influence, epsilon, fill_value, reduce_data, nprocs, segments)
    255         super(ImageContainerNearest, self).__init__(image_data, geo_def,
    256                                                     fill_value=fill_value,
--> 257                                                     nprocs=nprocs)
    258         self.radius_of_influence = radius_of_influence
    259         self.epsilon = epsilon

/srv/conda/envs/notebook/lib/python3.7/site-packages/pyresample/image.py in __init__(self, image_data, geo_def, fill_value, nprocs)
     59             geo_def = geo_def.freeze()
     60         if not isinstance(image_data, (np.ndarray, np.ma.core.MaskedArray)):
---> 61             raise TypeError('image_data must be either an ndarray'
     62                             ' or a masked array')
     63         elif ((image_data.ndim > geo_def.ndim + 1) or

TypeError: image_data must be either an ndarray or a masked array

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the latest pyresample documentation and the image.ImageContainerNearest entry point, then review the dask-related context in issue #148 and the reported dask.array example. Clarify the current xarray/dask support and document the supported behavior or limitations, including whether lazy resampling is expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.