MADIS netCDF to Pandas Dataframe: ValueError: iterator is too large
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Getting started with xarray 0.7 (python2.7 RHEL7.2 64bit) and I am having trouble converting a MADIS produced netCDF file to a pandas Dataframe. Here's the traceback
Traceback (most recent call last):
File "test.py", line 3, in <module>
df = nc.to_dataframe()
File "/usr/lib/python2.7/site-packages/xarray/core/dataset.py", line 1867, in to_dataframe
return self._to_dataframe(self.dims)
File "/usr/lib/python2.7/site-packages/xarray/core/dataset.py", line 1856, in _to_dataframe
for k in columns]
File "/usr/lib/python2.7/site-packages/xarray/core/variable.py", line 715, in expand_dims
expanded_data = ops.broadcast_to(self.data, tmp_shape)
File "/usr/lib/python2.7/site-packages/xarray/core/ops.py", line 65, in f
return getattr(eager_module, name)(data, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/numpy/lib/stride_tricks.py", line 115, in broadcast_to
return _broadcast_to(array, shape, subok=subok, readonly=True)
File "/usr/lib64/python2.7/site-packages/numpy/lib/stride_tricks.py", line 70, in _broadcast_to
op_flags=[op_flag], itershape=shape, order='C').itviews[0]
ValueError: iterator is too large
here's the code
import xarray as xr
nc = xr.open_dataset('20160430_1600.nc', decode_cf=False)
df = nc.to_dataframe()
Here's the netcdf file
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the supplied netCDF file and the xarray.open_dataset(..., decode_cf=False) and to_dataframe() calls. Start with Dataset.to_dataframe and its _to_dataframe path, then inspect Variable.expand_dims and the NumPy broadcast_to call shown in the traceback. Done means the dataset converts to a pandas DataFrame without the iterator-size error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100