Representing & checking Dataset schemas
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What would be the best way to canonically describe a dataset, which could be read by both humans and machines?
For example, frequently in our code we have docstrings which look something like:
def get_returns(security_ids):
"""
Retuns mega-dimensional dataset which gives recent returns for a set of
securities by:
- Date
- Return (raw / economic / smoothed / etc)
- Scaling (constant / risk_scaled)
- Span
- Hedged vs Unhedged
Dataset keys are security ids. All dimensions have coords.
"""
This helps when attempting to understand what code is doing while only reading it.
But this isn't consistent between docstrings and can't be read or checked by a machine.
Has anyone solved this problem / have any suggestions for resources out there?
Tangentially related to https://github.com/python/typing/issues/513 (but our issues are less about the type, dimension sizes, and more about the arrays within a dataset, their dimensions, and their names)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the proposed dataset docstring example and the related Python typing issue linked in the discussion. Determine whether the desired outcome is a schema representation, machine-checking support, or both; done would require an agreed canonical format and an implementation plan, neither of which this issue currently specifies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100