lincc-frameworks / lincc-frameworks/nested-pandas
Potential performance regression in `count_nested`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
Bug report
It was discovered in an LSDB workflow involving TESS, which has much larger nested structures than we are used to working with (~20000 observations per row), that count_nested was adding considerable memory pressure to a workflow that looked like this:
import lsdb
tess = lsdb.open_catalog("/astro/store/shire/hats/catalogs/tess/tess_lightcurve")
def drop_nans(df):
# 1. Drop the rows where lightcurve.sap_flux is NaN
# 2. Also remove all the resulting objects with no lightcurve points remaining
df["lightcurve.sap_flux"] = df["lightcurve.sap_flux"].astype(float)
return df.dropna(subset=["lightcurve.sap_flux"]).dropna(subset=["lightcurve"])
tess_filtered_nans = tess.map_partitions(drop_nans)
from nested_pandas.utils import count_nested
def count_points(pts):
# Asked to count `lightcurve`, this will add a column called `n_lightcurve`
return count_nested(pts, "lightcurve")
min_observations = 100
tess_filtered_nans = tess_filtered_nans.map_partitions(count_points).query(
f"n_lightcurve >= {min_observations}"
)
This was determined by stepping through and isolating the count_points method that wraps count_nested through map_partitions. When replaced by the recently added len approach, memory pressure was reduced by almost 8x:
def filter_lightcurves(nf):
return nf[nf["lightcurve"].len > 100]
The underlying implementation in count_nested should be similar, but something is potentially off here.
Traceback
FILL IN YOUR STACK TRACE HERE
Before submitting
Please check the following:
- I have described the situation in which the bug arose, including what code was executed, and any applicable data others will need to reproduce the problem.
- I have included information about my environment, including the version of this package (e.g.
nested_pandas.__version__) - I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the count_nested entry point and reproduce the provided LSDB/TESS map_partitions workflow, comparing it with the recently added len approach. Confirm that count_nested preserves its counting behavior without the reported memory pressure; the issue does not name a specific source file or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100