lincc-frameworks / lincc-frameworks/nested-pandas
Allow filtering via masking
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
I wanted to reproduce the following pandas workflow (all_forced_sources_w19 is flat pandas df with sources):
> '''Identify all columns that contain 'flag' '''
> flag_cols = [col for col in all_forced_sources_w19.columns if 'flag' in col.lower()]
>
> '''Exclude rows where any flag column is True'''
> flag_mask = ~(all_forced_sources_w19[flag_cols].any(axis=1)) # True where all flags are False
> all_forced_sources_w19_clean = all_forced_sources_w19[flag_mask]
The solution we found via `.query` was something like
> '''Identify flag columns'''
>flag_cols = [col for col in dia_object_lc_computed.diaObjectForcedSource.nest.fields if 'flag' in col.lower()]
>'''Build the condition string, e.g., "flag1 == False & flag2 == False & ..." '''
> query_str = " & ".join([f"diaObjectForcedSource.{col} == False" for col in flag_cols])
> '''Apply query directly to the nested column'''
> dia_object_lc_computed_filtered = dia_object_lc_computed.query(query_str)
I wish I was able to more directly mask, something like
> flag_mask = ~(dia_object_lc_computed.diaObjectForcedSource.nest.fields[flag_cols].any(axis=1))
> dia_object_lc_computed.diaObjectForcedSource = dia_object_lc_computed.diaObjectForcedSource[flag_mask]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the existing `.query` path and nested field access shown in the issue, then locate the entry point for masking nested data. Define how selecting `flag_cols` and applying a boolean row mask should behave, and verify that the direct masking workflow produces the same filtered rows as the demonstrated query workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100