apply_ufunc(dask='parallelized') output_dtypes for datasets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
When a Dataset has variables with different dtypes, there's no way to tell apply_ufunc that the same function applied to different variables will produce different dtypes:
ds1 = xarray.Dataset(data_vars={'a': ('x', [1, 2]), 'b': ('x', [3.0, 4.5])}).chunk()
ds2 = xarray.apply_ufunc(lambda x: x + 1, ds1, dask='parallelized', output_dtypes=[float])
ds2
<xarray.Dataset>
Dimensions: (x: 2)
Dimensions without coordinates: x
Data variables:
a (x) float64 dask.array<shape=(2,), chunksize=(2,)>
b (x) float64 dask.array<shape=(2,), chunksize=(2,)>
ds2.compute()
<xarray.Dataset>
Dimensions: (x: 2)
Dimensions without coordinates: x
Data variables:
a (x) int64 2 3
b (x) float64 4.0 5.5
Proposed solution
When the output is a dataset, apply_ufunc could accept either output_dtypes=[t] (if all output variables will have the same dtype) or output_dtypes=[{var1: t1, var2: t2, ...}]. In the example above, it would be output_dtypes=[{'a': int, 'b': float}].
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the apply_ufunc entry point and reproduce the Dataset example from the issue, focusing on dask='parallelized' handling of output_dtypes. The work is done when Dataset outputs accept per-variable dtype mappings while the existing shared-dtype form continues to work and the example preserves the intended dtypes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100