astronomy-commons / astronomy-commons/lsdb
read_hats filters argument should take PyArrow Expressions
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
**Feature request**
The documentation for `lsdb.read_hats` defers to the Pandas Parquet file reader for the format of `filters=` (via `**kwargs`). The documentation for `pd.read_parquet` describes its format, and also says that it won't work unless `engine=pyarrow`.
Parquet's `read_pandas` documentation describes this filtering syntax *and* the `pyarrow.compute.Expression` API, which allows for expressions like `pc.field('phot_variable_flag') == 'CONSTANT'`.
But using these filters with `lsdb.read_hats` results in errors. It would be valuable to say things like this:
```python
gaia3 = lsdb.read_hats(
'/data3/epyc/data3/hats/catalogs/gaia_dr3/gaia',
filters=[pc.field('phot_variable_flag') == 'CONSTANT'])
```
**Before submitting**
Please check the following:
- [X] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [X] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the lsdb.read_hats entry point and trace how its filters= kwargs reach the Pandas Parquet reader; compare that path with pd.read_parquet and the documented PyArrow Expression API. Verify the supplied pc.field('phot_variable_flag') expression works through lsdb.read_hats and filters the catalog without errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100