Structured numpy arrays, xarray and netCDF(4)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
I'm trying to use xarray as the underlying container for some data processing tasks. Part of the pipeline includes processing from non-standard/easily readable formats (e.g. ROS messages) to standard formats, e.g. netCDF(4). The data I tend to be working on is time series data that is structured, which maps pretty well to structured numpy arrays using dtype manipulations. And xarray lightly wraps numpy, and provides netCDF as a backend. However, the xarray implementation doesn't really expose this capability, supported in netCDF as 'compound data types', and in fact it fails when you try and write such a DataArray/Dataset to file (at _nc4_values_and_dtype).
So the question is, is this a reasonable feature/expectation from xarray (and thus you're receptive to contributions), or is this outside the goal/purpose (I should roll my own/use pandas/etc)?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing xarray/backends/netCDF4_.py, especially _nc4_values_and_dtype, and the linked NumPy structured-array and netCDF4 compound-type documentation. Determine whether structured arrays and compound data types fit xarray's intended scope; a contribution would need an agreed behavior and validation that DataArray or Dataset values can be written correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100