lincc-frameworks / lincc-frameworks/nested-pandas

dropna on a pyarrow float subcolumn silently keeps NaN rows

Open
#518 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

LSDB
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

NestedFrame.dropna(subset=["nested.field"]) does not drop NaN when the subcolumn is pyarrow-backed and the missing values are float NaN rather than Arrow nulls.
Reported by @drewoldag in https://discovery-alliance.slack.com/archives/C04610PQW9F/p1785974842086829, when they tried filtering tess_lightcurve on lightcurve.sap_flux with many NaN values which were not removed.

import numpy as np
import pandas as pd
import pyarrow as pa
from nested_pandas import NestedFrame

flux = pa.array([1.0, np.nan, 3.0, np.nan], type=pa.float64())
flat = pd.DataFrame({"flux": pd.arrays.ArrowExtensionArray(flux)}, index=[0, 0, 1, 1])
nf = NestedFrame({"id": [10, 11]}, index=[0, 1]).join_nested(flat, "lc") # a nested float column holding NaN

print(len(nf.dropna(subset=["lc.flux"])["lc"].nest.to_flat())) # 4, expected 2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided NestedFrame.dropna(subset=["lc.flux"]) reproduction and inspect the dropna path for nested pyarrow-backed subcolumns. The fix is done when float NaN rows are removed as expected while the resulting nested data retains the two valid rows; verify the example output after the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.