pydata / pydata/xarray

Representing & checking Dataset schemas

Open
#1,900 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-typing
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What would be the best way to canonically describe a dataset, which could be read by both humans and machines?

For example, frequently in our code we have docstrings which look something like:

def get_returns(security_ids):
    """
    Retuns mega-dimensional dataset which gives recent returns for a set of
        securities by:
    - Date
    - Return (raw / economic / smoothed / etc)
    - Scaling (constant / risk_scaled)
    - Span
    - Hedged vs Unhedged

    Dataset keys are security ids. All dimensions have coords.
    """

This helps when attempting to understand what code is doing while only reading it.
But this isn't consistent between docstrings and can't be read or checked by a machine.
Has anyone solved this problem / have any suggestions for resources out there?

Tangentially related to https://github.com/python/typing/issues/513 (but our issues are less about the type, dimension sizes, and more about the arrays within a dataset, their dimensions, and their names)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the proposed dataset docstring example and the related Python typing issue linked in the discussion. Determine whether the desired outcome is a schema representation, machine-checking support, or both; done would require an agreed canonical format and an implementation plan, neither of which this issue currently specifies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.