Automatic dtype encoding in to_netcdf
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Code Sample, a copy-pastable example if possible
Example from https://stackoverflow.com/questions/49053692/csv-to-netcdf-produces-nc-files-4x-larger-than-the-original-csv
import pandas as pd
import xarray as xr
import numpy as np
import os
# Create pandas DataFrame
df = pd.DataFrame(np.random.randint(low=0, high=10, size=(100000,5)),
columns=['a', 'b', 'c', 'd', 'e'])
# Make 'e' a column of strings
df['e'] = df['e'].astype(str)
# Save to csv
df.to_csv('df.csv')
# Convert to an xarray's Dataset
ds = xr.Dataset.from_dataframe(df)
# Save NetCDF file
ds.to_netcdf('ds.nc')
# Compute stats
stats1 = os.stat('df.csv')
stats2 = os.stat('ds.nc')
print('csv=',str(stats1.st_size))
print('nc =',str(stats2.st_size))
print('nc/csv=',str(stats2.st_size/stats1.st_size))
The result:
>>> csv = 1688902 bytes
>>> nc = 6432441 bytes
>>> nc/csv = 3.8086526038811015
Problem description
NetCDF can store numerical data, as well as some other data, such as categorical data, in a much more efficient way than CSV, do to it's ability to store numbers (integers, limited precision floats) in smaller encodings (e.g. 8 bit integers), as well as it's ability to compress data using zlib.
The answers in the stack exchange link at the top of the page give some examples of how this can be done. The second one is particularly useful, and it would be nice if xarray provided an encoding={'dtype': 'auto'} option to automatically select the most compact dtype able to store a given DataArray before saving the file.
Expected Output
NetCDF from should be equal to or smaller than a CSV full of numerical data in most cases.
Output of xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7 | packaged by conda-forge | (default, Nov 21 2018, 03:09:43)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.18.0-15-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: en_AU.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.2
xarray: 0.11.3
pandas: 0.24.1
netCDF4: 1.4.2
h5netcdf: 0.6.2
h5py: 2.9.0
dask: 1.0.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the xarray Dataset.to_netcdf entry point and reproduce the provided pandas/NumPy dataframe example. Define and test the automatic dtype-selection behavior, including which values and encodings must remain valid; done means the generated NetCDF is no larger in the stated numerical case while preserving the data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100