astronomy-commons / astronomy-commons/lsdb-server

Apply filters to non-selected columns

Open
#8 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
PR merge metrics
No merged PRs in 30d

Description

With PyArrow and Pandas, a user can select a set of columns to return and filter by a column that is not in this set. For example:

```python
import pandas as pd

frame = pd.read_parquet('Npix=0.parquet', columns=['ra', 'dec'], filters=[('yMeanPSFMag', '<', 15)])
```

However, when I try to do the same with lsdb.read_hats and the HATS server, I always get an empty DataFrame.

Could we make column selection a two-step process, where we first load the columns specified both by columns and filters, apply the filter, and then remove the columns that appeared only in filters?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the lsdb.read_hats entry point and trace how the columns and filters are passed to the HATS server and applied. Reproduce the example with columns ['ra', 'dec'] and the yMeanPSFMag filter, then verify that filtering works and filter-only columns are absent from the returned DataFrame.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, rust
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.