lincc-frameworks / lincc-frameworks/nested-pandas

Pandas Function Extension Megatracker

Open
#295 0 comments 1 reaction 0 assignees View on GitHub
enhancement interface
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

Nested-Pandas augments a number of Pandas functions with additional behavior, but there are many additional functions we haven't yet modified. This issue serves as a large tracker for functions that could/should be extended to nested-pandas, along with a brief idea of how they might be extended to work with nested columns

This list is not comprehensive.

Potential Pandas Functions to Extend
-------------------------------------
- [x] [DataFrame.max()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.max.html): We can add support for returning quick global max values for nested float/int sub-columns as well **_Impact: High_** #314
- [x] [DataFrame.min()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.min.html): We can add support for returning quick global min values for nested float/int sub-columns as well **_Impact: High_** #307
- [x] [DataFrame.explode()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.explode.html): Allow explode to work on nested columns, unpacking them to a new flat frame with base column values repeated on index **_Impact: High_**, separate issue #309
- [x] [DataFrame.describe()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html): We can add support for returning full statistical output for nested sub-columns **_Impact: High_** #322
- [x] [DataFrame.fillna()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html): Allow fillna to also fill nans in nested sub columns would be very useful I think **_Impact: High_** https://github.com/lincc-frameworks/nested-pandas/pull/335
- [ ] [DataFrame.mean()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.mean.html): We can add support for returning quick global mean values for nested float/int sub-columns as well **_Impact: High_**
- [ ] [DataFrame.median()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.mean.html): We can add support for returning quick global median values for nested float/int sub-columns as well **_Impact: High_**
- [ ] [DataFrame.mode()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.mean.html): We can add support for returning quick global mode values for nested float/int sub-columns as well **_Impact: High_**
- [ ] [DataFrame.select_dtypes()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.select_dtypes.html): We should add some support for nested column dtypes, which differ from base columns due to the pyarrow typing a la describe. **_Impact: Medium_**
- [ ] [pandas.read_csv()](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html): If we more appropriately handle csv writing, then we should provide a thin wrapper for reading as well. **_Impact: Medium_**
- [ ] [DataFrame.to_csv()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html): Nested columns likely don't work well when serialized to csv, so we should potentially modify this to produce multiple output csv files, one base and one per nested structure. **_Impact: Medium_**
- [ ] [DataFrame.where()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.where.html): Could operate on nested sub-columns and generate a 2d mask (list of mask arrays) that corresponds to the dimensionality of that nested structure. **_Impact: Low_**
- [ ] [DataFrame.mask()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.mask.html): Could directly mask nested subcolumns. **_Impact: Low_**
- [ ] [DataFrame.apply()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html): reduce was thought of as a replacement for this, but it's such a notable function in pandas we should think about how to support it properly, perhaps even renaming reduce to apply and just supporting nested columns directly in apply would be best? **_Impact: ? (What would we do with apply that we can't do with reduce?)_**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the unchecked Pandas functions in this tracker and the completed extensions linked for DataFrame.max(), min(), explode(), describe(), and fillna(). Narrow the work to one function and define its behavior for nested columns; this tracker is complete only when each selected extension has an agreed scope and implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.