[Question] Efficient shortcut for unstacking only parts of dimension?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Hi all, is there an efficient way to unstack only parts of a MultiIndex?
Consider for example the following array:
<xarray.Dataset>
Dimensions: (observations: 17525)
Coordinates:
* observations (observations) MultiIndex
- subtissue (observations) object 'Skin_Sun_Exposed_Lower_leg' ... 'Thyroid'
- individual (observations) object 'GTEX-111FC' ... 'GTEX-ZZPU'
- gene (observations) object 'ENSG00000140400' ... 'ENSG00000174233'
- end (observations) object '5' '5' '5' ... '3' '3'
Data variables:
fraser_min_pval (observations) float64 dask.array<chunksize=(17525,), meta=np.ndarray>
fraser_min_minus_log10_pval (observations) float64 dask.array<chunksize=(17525,), meta=np.ndarray>
Here, I have a MultiIndex observations=["subtissue", "individual", "gene", "end"].
However, I would like to have end in its own dimension.
Currently, I have to do the following to solve this issue:
xrds.unstack("observations").stack(observations=["subtissue", "individual", "gene",])
However, this seems quite inefficient and introduces NaN's.
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1062.1.2.el7.x86_64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.7.3
xarray: 0.15.0
pandas: 1.0.0
numpy: 1.17.5
scipy: 1.4.1
netCDF4: 1.5.3
pydap: None
h5netcdf: 0.7.4
h5py: 2.10.0
Nio: None
zarr: 2.4.0
cftime: 1.0.4.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.10.1
distributed: 2.10.0
matplotlib: 3.1.3
cartopy: None
seaborn: 0.10.0
numbagg: None
setuptools: 45.1.0.post20200119
pip: 20.0.2
conda: None
pytest: 5.3.5
IPython: 7.12.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading xarray's stack and unstack entry points and how they handle pandas MultiIndex dimensions. Investigate whether a partial unstack can preserve the remaining observations without introducing NaNs; done means an efficient supported operation that places end in its own dimension while retaining the other levels.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100