ga4gh / ga4gh/analytics-dashboard-api

Classification service — classify staged records against production

Open
#150 0 comments 0 reactions 1 assignee Claimed by @dashrath-chauhan View on GitHub
Dashboard dev enhancement
Dominant language
Jupyter Notebook
Stars
4
Forks
0
Avg merge
3d 8h
Merged PRs (30d)
2

Description

Two-database session routing is in place. After each ingestion pull, records land in staging with no comparison against production. This issue implements the classification service defined in doc/epmc-data-provenance-plan.md.

**Two design decisions confirmed before drafting this issue:**

**Full pull every run** — the ingestion pipeline always pulls all records from EPMC. Staging is intentionally an append-only historical record (one set of rows per epmc_id per ingestion run). Staging data is rotated periodically to a data lake. No upsert — insert only.
**ingestion_id is the navigation key** — version on child tables (pmc_articles, citations, pmc_authors, etc.) is always 1 and was never used. All "latest record" queries use ingestion_id. version on the ingestion table itself is meaningful.

**Classification Flow**
The service runs after every ingestion pull. It reads only the rows written in the current ingestion_id, compares each against production, and writes results to pmc_review and audit_log.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.