galaxyproject / galaxyproject/brc-analytics
Add P. vivax pangenome bundle to BRC-analytics (first deliverable: Pangenome tab on organism page)
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 11
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 16
Description
## What this is
Add a **per-species pangenome bundle** to BRC-analytics, hosted as a new "Pangenome" tab on the organism detail page. First bundle: **`plasmodium-vivax-v1`** — 8 *P. vivax* strain assemblies anchored on PvP01, with derived ortholog tables, codon MSAs, ML trees, HyPhy BUSTED selection scans, and MalariaGEN cohort variant projections.
Full analysis lives at **https://github.com/nekrut/Pv4-pangenome** (public). Detailed deployment sketch in [`v3/writeup/BRC_DEPLOYMENT.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/BRC_DEPLOYMENT.md). Six analysis docs ([`PANGENOME.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/PANGENOME.md), [`MULTIZ.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MULTIZ.md), [`ORTHOLOGY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/ORTHOLOGY.md), [`MSA_HYPHY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MSA_HYPHY.md), [`MALARIAGEN_VCF_PROJECTION.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MALARIAGEN_VCF_PROJECTION.md), [`MICROSYNTENY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MICROSYNTENY.md)) describe each layer with file paths + sizes + recreate recipes.
## Where this lands in the UI
A new **"Pangenome" tab** on the organism detail page for *P. vivax* (taxon 5855). No new top-level route, no separate `PangenomeView`. The tab's sections:
- **Bundle downloads** — the 8-way PGGB graph (GFA + odgi)
- **Member assemblies** — table of the 8 strains with links to each assembly's UCSC hub (where the alignment, annotation, selection, and cohort-VCF tracks live)
- **Orthogroup browser** — gene name / sequence lookup → per-OG panel with MSA + tree + BUSTED result
- **Cohort VCF** — MalariaGEN 1,895-sample VCFs download + per-assembly hub link
Sibling route for UCSC click-through: `/organisms/5855/pangenome/orthogroup/{gene_id}`.
The organism-page tabbed layout was set up by PRs #1261, #1263, #1274, #1277, #1278 (May 2026). The Pangenome tab is one more entry in that pattern.
## 5 data blocks → deployment
All file paths and sizes are in [`BRC_DEPLOYMENT.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/BRC_DEPLOYMENT.md). Summary:
| # | Block | Files | Where it lands |
|---|-------------------------------------|-------------------------------|---------------------------------------------------------------------------|
| 1 | Pangenome graph (PGGB) | 3 (GFA + odgi + layout) | Download links on the Pangenome tab. No UCSC track in v1. |
| 2 | Pairwise chains + multi-way MAFs | 56 chains + 8 bigMafs | UCSC composite `brc_pangenome_align` on each of 8 assembly hubs. |
| 3 | Cross-strain annotations + ortholog table | 28 GFFs + ortholog/family tables + 32 BigBeds | UCSC composite `brc_pangenome_annot` on each hub + tables in tab. |
| 4 | Per-OG MSAs + ML trees + HyPhy BUSTED | ~5,800 per-OG bundles + 3 PvP01 BigBeds | Orthogroup browser in tab (lookup → per-OG panel) + UCSC composite `brc_pangenome_select` on PvP01 hub. |
| 5 | MalariaGEN cohort VCF projections | 16 source + 7 lifted VCFs | UCSC composite `brc_pangenome_cohort` (`vcfTabix`) on each hub. |
## Hosting
| Tier | What | Where |
|------|------|-------|
| Git | Small text (chains, ortholog table, family table, MSA archives, hub manifests) | https://github.com/nekrut/Pv4-pangenome |
| Dropbox | Heavy binary (cohort VCFs, AXTs, multiz MAFs, bigMafs, PGGB .og + .lay) | https://www.dropbox.com/scl/fo/gx1mta4adubja4bsxxmgm/AOni3YRX8TS1E-saUblX-eo?rlkey=0ksw8a5hhkxhy3sljqyycr7bu&dl=0 — 339 GB across 369 files |
| TACC | Long-term home for the per-OG MSAs + trees + BUSTED JSONs that the orthogroup browser fetches on demand | To be provisioned by BRC |
| UCSC hub | Track binaries (bigMaf, BigBed, chains, vcfTabix) | `hgdownload.soe.ucsc.edu/hubs/BRC/pangenome_plasmodium_vivax_v1/` |
MD5 manifest for the heavy files: `v3/writeup/LARGE_FILES_DROPBOX.tsv` (132 entries).
## Catalog entry sketch
```yaml
# catalog/source/pangenomes.yml
- id: plasmodium-vivax-v1
species_taxonomy_id: 5855
version: "2026-05"
reference_anchor: GCA_900093555.2 # PvP01
member_assemblies:
- GCA_900093555.2 # PvP01 (anchor)
- GCA_000002415.2 # Sal-I (note: catalogued as GCF_000002415.2; translate in build step)
- GCA_914969965.1 # PvW1
- GCA_949152365.1 # PAM
- GCA_003402215.1 # PvSY56
- GCA_900093545.1 # PvT01
- GCA_900093535.1 # PvC01
- GCA_040114635.1 # MHC087
```
Track inventory auto-populated by the build step from a `manifest.json` on TACC / datacache.
Schema additions to `catalog/py_package/catalog_build/schema/pangenomes.yaml` (new file):
- `Pangenome.graph` (GFA1 PangenomeArtifact)
- `Pangenome.chain_files[]`
- `Pangenome.multiz_alignments[]`
- `Pangenome.merged_annotations[]`
- `Pangenome.ortholog_table`
- `Pangenome.family_table`
- `Pangenome.cds_alignments[]`
- `Pangenome.gene_trees`
- `Pangenome.cohort_vcf_source`
- `Pangenome.cohort_vcf_projections[]`
- `Pangenome.selection_tracks[]` (BUSTED results)
## UCSC hub layout
```
hgdownload.soe.ucsc.edu/hubs/BRC/pangenome_plasmodium_vivax_v1/
├── hub.txt
├── genomes.txt
└── {ACC}/ # one dir per assembly (8 total)
├── trackDb.txt # 4 composites
├── {hinge}.multiz.maf.bb # Block 2 — bigMaf
├── chains/{ACC}_to_{ACC_T}.chain.gz # Block 2 — 7 per assembly
├── annot_from_{anchor}.bb # Block 3 — 4 per assembly
├── selection_{strict,relaxed}.bb # Block 4 — PvP01 only
├── orthogroup_membership.bb # Block 4 — PvP01 only
└── Pv4_cohort_on_{ACC}.vcf.gz + .tbi # Block 5 — 1 per assembly
```
Each gene feature in the annotation BigBed tracks gets a `url` attribute in `trackDb.txt` of the form `https://brc-analytics.org/organisms/5855/pangenome/orthogroup/$$` so clicking a gene in the UCSC browser lands on the per-OG panel in BRC-analytics.
## 4-PR sequence
| PR | Adds | Risk |
|----|------|------|
| 1 | Schema additions (LinkML in `catalog/py_package/catalog_build/schema/pangenomes.yaml`). Empty `catalog/source/pangenomes.yml`. `build_pangenomes` step → `catalog/output/pangenomes.json` + `assembly-artifacts.json`. GCA↔GCF translation for Sal-I. | Low — purely additive |
| 2 | Ingest Pv-v1 artifacts from Dropbox → TACC + datacache. `publish_ucsc_hub` step generating per-assembly `trackDb.txt` + `hub.txt` + `genomes.txt`. Concat + tabix-index the cohort VCFs. Rsync to `hgdownload.soe.ucsc.edu/hubs/BRC/`. | Low — static-file generation |
| 3 | "Pangenome" tab on the *P. vivax* organism page (`/organisms/5855`) with the 4 sections above. Orthogroup detail route at `/organisms/5855/pangenome/orthogroup/{id}`. MSAViewer.js + phylotree.js modals. HyPhy-Vision deep-link (pending `veg/hyphy-vision#892`). | Medium — UI |
| 4 | `POPULATION_GENOMICS` + `SELECTION_ANALYSIS` workflow categories. IWC workflow registration for `pangenome-build-and-project` + `hyphy-selection-screen-from-ortholog-table` so users can re-run the pipeline on other panels. | Medium — Galaxy integration |
Mergeable independently. Suggested cadence: PR 1 first, then PR 2 + PR 3 in parallel, PR 4 last.
## What developers need from me
1. **GitHub repo** (public): https://github.com/nekrut/Pv4-pangenome
2. **Dropbox folder** (world-readable): https://www.dropbox.com/scl/fo/gx1mta4adubja4bsxxmgm/AOni3YRX8TS1E-saUblX-eo?rlkey=0ksw8a5hhkxhy3sljqyycr7bu&dl=0
3. **The 6 doc files in `v3/writeup/`** — [`BRC_DEPLOYMENT.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/BRC_DEPLOYMENT.md) is the deployment sketch; the other 5 ([`PANGENOME.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/PANGENOME.md), [`MULTIZ.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MULTIZ.md), [`ORTHOLOGY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/ORTHOLOGY.md), [`MSA_HYPHY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MSA_HYPHY.md), [`MALARIAGEN_VCF_PROJECTION.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MALARIAGEN_VCF_PROJECTION.md)) describe each block in detail with file paths + recreate recipes
4. **MD5 manifest**: `v3/writeup/LARGE_FILES_DROPBOX.tsv` (132 entries)
## What I need from developers
- A staging area on `datacache.galaxyproject.org/brc/data/pangenomes/plasmodium-vivax-v1/` and on TACC to rsync the Dropbox contents into
- A `manifest.json` generator (probably an `aws s3 ls` equivalent on TACC) so `build_pangenomes` can auto-populate track lists
- The 4-PR sequence implemented
## Related issues
- veg/hyphy-vision#892 — CORS + deep-link support for `?json=` queries (gates PR 3's per-OG panel deep-link)
- The old version of this issue body had a more elaborate `PangenomeView` design — that's been replaced by the simpler "tab on organism page" approach reflected here.
## Companion strategic context
Two strategic issues drafted earlier for the `BRC-research` repo describe the broader gap-filling roadmap and the primary-dataset ingest map. The `v3/writeup/BRC_DEPLOYMENT.md` doc in the Pv4-pangenome repo is the implementation slice for the first deliverable in that roadmap.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with v3/writeup/BRC_DEPLOYMENT.md and the four-PR sequence, then inspect the existing organism-page tab pattern from PRs #1261, #1263, #1274, #1277, and #1278. For the initial schema slice, read catalog/py_package/catalog_build/schema/pangenomes.yaml and catalog/source/pangenomes.yml; done requires the catalog outputs and subsequent UI, hosting, and workflow pieces to be implemented as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- build-system, data, devops, full-stack
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100