astronomy-commons / astronomy-commons/lsdb

read_hats filters argument should take PyArrow Expressions

Open
#652 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

**Feature request**

The documentation for `lsdb.read_hats` defers to the Pandas Parquet file reader for the format of `filters=` (via `**kwargs`). The documentation for `pd.read_parquet` describes its format, and also says that it won't work unless `engine=pyarrow`.

Parquet's `read_pandas` documentation describes this filtering syntax *and* the `pyarrow.compute.Expression` API, which allows for expressions like `pc.field('phot_variable_flag') == 'CONSTANT'`.

But using these filters with `lsdb.read_hats` results in errors. It would be valuable to say things like this:

```python
gaia3 = lsdb.read_hats(
'/data3/epyc/data3/hats/catalogs/gaia_dr3/gaia',
filters=[pc.field('phot_variable_flag') == 'CONSTANT'])
```

**Before submitting**
Please check the following:

- [X] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [X] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the lsdb.read_hats entry point and trace how its filters= kwargs reach the Pandas Parquet reader; compare that path with pd.read_parquet and the documented PyArrow Expression API. Verify the supplied pc.field('phot_variable_flag') expression works through lsdb.read_hats and filters the catalog without errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.