astronomy-commons / astronomy-commons/lsdb

Add example of pyarrow `filters` to the docs

Open Beginner friendly
#1,306 2 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

The docs page for [row filtering](https://docs.lsdb.io/en/latest/tutorials/row_filtering.html) include examples of `Catalog.query(...)` but we are missing instructions on how to use pyarrow filters. E.g.:

```python
gaia_dr3 = lsdb.open_catalog(
"s3://stpubdata/gaia/gaia_dr3/public/hats",
columns=["pmra", "pmdec", "radial_velocity", "parallax"],
filters=[("pm", "<", max_pm), ("parallax", ">", 0.0)],
)
```

We should:
- Highlight that these filters are very efficient for S3 cloud reads (since they are applied server-side).
- Include a pointer to the official [ParquetDataset](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetDataset.html) docs (see `filters` kwarg) since the syntax for the filters can be unintuitive for users.

Contributor guide

No contributing guide indexed for this repository

Research direction

Open the row filtering docs page and review its existing Catalog.query examples before adding the requested pyarrow filters example. Done means the page explains server-side efficiency for S3 cloud reads and links to the official ParquetDataset documentation for the filters syntax.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.