pydata / pydata/xarray

Provide protocols for creating structural subtypes of DataArray/Dataset

Open
#6,462 5 comments 11 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement topic-typing
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?

I frequently find myself wanting to annotate functions in terms of xarray objects that adhere to a particular schema. Given that a dataset's adherence to a schema is a matter of its structure/contents, it is unnatural to try to describe a schema as a subtype of xr.Dataset (or DataArray) (i.e. a type-checker ought not care that a dataset is an instance of a specific subclass of Dataset).

Describe the solution you'd like

Instead, it would be ideal to define a schema as a Protocol (structural subtype) of xr.Dataset. Unfortunately, one cannot subclass a normal class to create a protocol.

Thus, I am proposing that xarray provide Protocol-based descriptions of DataArray and Dataset so that users can describe schemas as structural subtypes of these classes. E.g.

from typing import Protocol

from xarray import DataArray
from xarray.typing import DatasetProtocol

class ClimateData(DatasetProtocol, Protocol):
    lat: DataArray
    lon: DataArray
    temp: DataArray
    precip: DataArray

def process_climate_data(ds: ClimateData):
    ds.banana  # type checker flags as unknown attribute
    ds.temp  # type checker sees "DataArray" (as informed by ClimateData)
    ds.sel(lat=1.0)  # type checker sees `Dataset` (as informed by `DatasetProtocol`)

The contents of DatasetProtocol would essentially look like a modified type stub for xarray.Dataset so the implementation details are relatively simple, I believe.

Describe alternatives you've considered

Creating a strict subtype of Dataset is not ideal for a few reasons:

  1. Static type checkers would then expect to see that datasets must derive from that particular subclass, which is generally not the case.
  2. The annotations / design of xarray.Dataset is too broad for describing a schema. E.g. the presence of __getattr__ prevents type checkers from flagging access to non-existent data variables and coordinates during static analysis. DatasetProtocol would need to be designed to be less permissive than this.
Additional context

Hopefully this could be leveraged by the likes of xarray-schema so that xarray schemas can be used to provide both runtime and static validation capabilities.

I'd love to get feedback on this, and would be happy to open a PR if xarray devs are willing to weigh in on the design of these protocols.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing xarray typing surface and the Dataset/DataArray annotations, then review PEP 544 and the proposed xarray.typing.DatasetProtocol example. Define what the protocols must expose and how schema protocols should narrow dataset attributes while retaining Dataset methods. Done means users can declare structural schema subtypes and type checkers recognize known attributes while rejecting unknown ones.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.