lincc-frameworks / lincc-frameworks/nested-pandas

Allow filtering via masking

Open
#273 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement interface LSDB
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

I wanted to reproduce the following pandas workflow (all_forced_sources_w19 is flat pandas df with sources):

> '''Identify all columns that contain 'flag' '''
> flag_cols = [col for col in all_forced_sources_w19.columns if 'flag' in col.lower()]
>
> '''Exclude rows where any flag column is True'''
> flag_mask = ~(all_forced_sources_w19[flag_cols].any(axis=1)) # True where all flags are False
> all_forced_sources_w19_clean = all_forced_sources_w19[flag_mask]

The solution we found via `.query` was something like

> '''Identify flag columns'''
>flag_cols = [col for col in dia_object_lc_computed.diaObjectForcedSource.nest.fields if 'flag' in col.lower()]
>'''Build the condition string, e.g., "flag1 == False & flag2 == False & ..." '''
> query_str = " & ".join([f"diaObjectForcedSource.{col} == False" for col in flag_cols])
> '''Apply query directly to the nested column'''
> dia_object_lc_computed_filtered = dia_object_lc_computed.query(query_str)

I wish I was able to more directly mask, something like

> flag_mask = ~(dia_object_lc_computed.diaObjectForcedSource.nest.fields[flag_cols].any(axis=1))
> dia_object_lc_computed.diaObjectForcedSource = dia_object_lc_computed.diaObjectForcedSource[flag_mask]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the existing `.query` path and nested field access shown in the issue, then locate the entry point for masking nested data. Define how selecting `flag_cols` and applying a boolean row mask should behave, and verify that the direct masking workflow produces the same filtered rows as the demonstrated query workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.