astronomy-commons / astronomy-commons/lsdb

Update pipeline for PPDB HATS records

Open
#1,297 0 comments 0 reactions 1 assignee Assigned to @olivialynn View on GitHub
blocked dataset enhancement
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

Implement an update pipeline that applies Rubin APDB replica update records to PPDB catalogs stored in HATS, so the HATS representation stays aligned with Observatory PPDB semantics without requiring full reimport for every change.

**User Story**
As a scientist or platform user querying PPDB through LSDB/HATS, I want that PPDB results in HATS remain scientifically correct, current, and comparable to Observatory-backed sources.

**Scope**
The initial implementation should support the [update record types currently represented in APDB](https://github.com/lsst/dax_apdb/blob/main/python/lsst/dax/apdb/apdbUpdateRecord.py) that dont touch solar system objects including:
- reassignment of DiaSource to DiaObject
- reassignment of DiaSource to SSObject - I suggest deleting these records for the time being, as we are not dealing with SS yet
- withdrawal of DiaSource
- withdrawal of DiaForcedSource
- closing DiaObject validity
- updating DiaObject nDiaSources

**Acceptance Criteria**
- A HATS update pipeline exists that consumes APDB update records and applies them in deterministic order.
- Applying the same ordered update record batch twice is safe or explicitly prevented with documented behavior.
- Updated HATS output passes validation against a reference BigQuery subset for representative cases.
- Publish procedure exists so that consumers can identify the latest promoted PPDB HATS version.
- Rollback procedure is documented for a bad update batch.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.