astronomy-commons / astronomy-commons/lsdb-server
Apply filters to non-selected columns
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
With PyArrow and Pandas, a user can select a set of columns to return and filter by a column that is not in this set. For example:
```python
import pandas as pd
frame = pd.read_parquet('Npix=0.parquet', columns=['ra', 'dec'], filters=[('yMeanPSFMag', '<', 15)])
```
However, when I try to do the same with lsdb.read_hats and the HATS server, I always get an empty DataFrame.
Could we make column selection a two-step process, where we first load the columns specified both by columns and filters, apply the filter, and then remove the columns that appeared only in filters?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the lsdb.read_hats entry point and trace how the columns and filters are passed to the HATS server and applied. Reproduce the example with columns ['ra', 'dec'] and the yMeanPSFMag filter, then verify that filtering works and filter-only columns are absent from the returned DataFrame.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, rust
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100