pydata / pydata/xarray

Structured numpy arrays, xarray and netCDF(4)

Open
#1,626 6 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contrib-help-wanted topic-backends
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

I'm trying to use xarray as the underlying container for some data processing tasks. Part of the pipeline includes processing from non-standard/easily readable formats (e.g. ROS messages) to standard formats, e.g. netCDF(4). The data I tend to be working on is time series data that is structured, which maps pretty well to structured numpy arrays using dtype manipulations. And xarray lightly wraps numpy, and provides netCDF as a backend. However, the xarray implementation doesn't really expose this capability, supported in netCDF as 'compound data types', and in fact it fails when you try and write such a DataArray/Dataset to file (at _nc4_values_and_dtype).

So the question is, is this a reasonable feature/expectation from xarray (and thus you're receptive to contributions), or is this outside the goal/purpose (I should roll my own/use pandas/etc)?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing xarray/backends/netCDF4_.py, especially _nc4_values_and_dtype, and the linked NumPy structured-array and netCDF4 compound-type documentation. Determine whether structured arrays and compound data types fit xarray's intended scope; a contribution would need an agreed behavior and validation that DataArray or Dataset values can be written correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.