astronomy-commons / astronomy-commons/lsdb

Non-fine filtering on margin catalogs results in strange crossmatch/join behavior

Open
#1,328 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

If a user performs a non-fine filter on a margin catalog, the resulting margin has very unexpected properties where points a healpix pixel width away from the search area is included, which results in strange crossmatches.

This is because to filter a margin to include all margin points within an area, we need to include the neighboring healpix pixels. With fine filtering, any points outside of the search region in neighboring pixels are removed, but without fine filtering, these are still included, which can be quite far away from the original search region.

The options for fixing this I can think of are:
- Make non-fine filtering ignore any margins pixels outside of the search region. This would solve this case, but would also lead to us missing potential matches.
- Make non-fine filtering drop the margin. This would cause us to lose even more potential matches, but would warn the user that they might be missing matches. Though the behavior of dropping a margin unexpectedly could be confusing.
- Make non-fine margin filtering do a fine pixel filter of the catalogs search region. This would make sure we catch all matches and the margin filter would contain only the points that exist in the main catalog, which seems sensible.

The third option is probably my favorite.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the non-fine margin filtering path and how neighboring healpix pixels are selected, then compare it with fine filtering for the catalog search region. The issue is resolved when non-fine margin filtering preserves potential matches without retaining points far outside the search area, and the resulting crossmatches behave as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.