galaxyproject / galaxyproject/brc-analytics

Add P. vivax pangenome bundle to BRC-analytics (first deliverable: Pangenome tab on organism page)

Open
#1,279 18 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
7
Forks
11
Avg merge
2d 12h
Merged PRs (30d)
16

Description

## What this is

Add a **per-species pangenome bundle** to BRC-analytics, hosted as a new "Pangenome" tab on the organism detail page. First bundle: **`plasmodium-vivax-v1`** — 8 *P. vivax* strain assemblies anchored on PvP01, with derived ortholog tables, codon MSAs, ML trees, HyPhy BUSTED selection scans, and MalariaGEN cohort variant projections.

Full analysis lives at **https://github.com/nekrut/Pv4-pangenome** (public). Detailed deployment sketch in [`v3/writeup/BRC_DEPLOYMENT.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/BRC_DEPLOYMENT.md). Six analysis docs ([`PANGENOME.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/PANGENOME.md), [`MULTIZ.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MULTIZ.md), [`ORTHOLOGY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/ORTHOLOGY.md), [`MSA_HYPHY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MSA_HYPHY.md), [`MALARIAGEN_VCF_PROJECTION.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MALARIAGEN_VCF_PROJECTION.md), [`MICROSYNTENY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MICROSYNTENY.md)) describe each layer with file paths + sizes + recreate recipes.

## Where this lands in the UI

A new **"Pangenome" tab** on the organism detail page for *P. vivax* (taxon 5855). No new top-level route, no separate `PangenomeView`. The tab's sections:

- **Bundle downloads** — the 8-way PGGB graph (GFA + odgi)
- **Member assemblies** — table of the 8 strains with links to each assembly's UCSC hub (where the alignment, annotation, selection, and cohort-VCF tracks live)
- **Orthogroup browser** — gene name / sequence lookup → per-OG panel with MSA + tree + BUSTED result
- **Cohort VCF** — MalariaGEN 1,895-sample VCFs download + per-assembly hub link

Sibling route for UCSC click-through: `/organisms/5855/pangenome/orthogroup/{gene_id}`.

The organism-page tabbed layout was set up by PRs #1261, #1263, #1274, #1277, #1278 (May 2026). The Pangenome tab is one more entry in that pattern.

## 5 data blocks → deployment

All file paths and sizes are in [`BRC_DEPLOYMENT.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/BRC_DEPLOYMENT.md). Summary:

| # | Block | Files | Where it lands |
|---|-------------------------------------|-------------------------------|---------------------------------------------------------------------------|
| 1 | Pangenome graph (PGGB) | 3 (GFA + odgi + layout) | Download links on the Pangenome tab. No UCSC track in v1. |
| 2 | Pairwise chains + multi-way MAFs | 56 chains + 8 bigMafs | UCSC composite `brc_pangenome_align` on each of 8 assembly hubs. |
| 3 | Cross-strain annotations + ortholog table | 28 GFFs + ortholog/family tables + 32 BigBeds | UCSC composite `brc_pangenome_annot` on each hub + tables in tab. |
| 4 | Per-OG MSAs + ML trees + HyPhy BUSTED | ~5,800 per-OG bundles + 3 PvP01 BigBeds | Orthogroup browser in tab (lookup → per-OG panel) + UCSC composite `brc_pangenome_select` on PvP01 hub. |
| 5 | MalariaGEN cohort VCF projections | 16 source + 7 lifted VCFs | UCSC composite `brc_pangenome_cohort` (`vcfTabix`) on each hub. |

## Hosting

| Tier | What | Where |
|------|------|-------|
| Git | Small text (chains, ortholog table, family table, MSA archives, hub manifests) | https://github.com/nekrut/Pv4-pangenome |
| Dropbox | Heavy binary (cohort VCFs, AXTs, multiz MAFs, bigMafs, PGGB .og + .lay) | https://www.dropbox.com/scl/fo/gx1mta4adubja4bsxxmgm/AOni3YRX8TS1E-saUblX-eo?rlkey=0ksw8a5hhkxhy3sljqyycr7bu&dl=0 — 339 GB across 369 files |
| TACC | Long-term home for the per-OG MSAs + trees + BUSTED JSONs that the orthogroup browser fetches on demand | To be provisioned by BRC |
| UCSC hub | Track binaries (bigMaf, BigBed, chains, vcfTabix) | `hgdownload.soe.ucsc.edu/hubs/BRC/pangenome_plasmodium_vivax_v1/` |

MD5 manifest for the heavy files: `v3/writeup/LARGE_FILES_DROPBOX.tsv` (132 entries).

## Catalog entry sketch

```yaml
# catalog/source/pangenomes.yml
- id: plasmodium-vivax-v1
species_taxonomy_id: 5855
version: "2026-05"
reference_anchor: GCA_900093555.2 # PvP01
member_assemblies:
- GCA_900093555.2 # PvP01 (anchor)
- GCA_000002415.2 # Sal-I (note: catalogued as GCF_000002415.2; translate in build step)
- GCA_914969965.1 # PvW1
- GCA_949152365.1 # PAM
- GCA_003402215.1 # PvSY56
- GCA_900093545.1 # PvT01
- GCA_900093535.1 # PvC01
- GCA_040114635.1 # MHC087
```

Track inventory auto-populated by the build step from a `manifest.json` on TACC / datacache.

Schema additions to `catalog/py_package/catalog_build/schema/pangenomes.yaml` (new file):

- `Pangenome.graph` (GFA1 PangenomeArtifact)
- `Pangenome.chain_files[]`
- `Pangenome.multiz_alignments[]`
- `Pangenome.merged_annotations[]`
- `Pangenome.ortholog_table`
- `Pangenome.family_table`
- `Pangenome.cds_alignments[]`
- `Pangenome.gene_trees`
- `Pangenome.cohort_vcf_source`
- `Pangenome.cohort_vcf_projections[]`
- `Pangenome.selection_tracks[]` (BUSTED results)

## UCSC hub layout

```
hgdownload.soe.ucsc.edu/hubs/BRC/pangenome_plasmodium_vivax_v1/
├── hub.txt
├── genomes.txt
└── {ACC}/ # one dir per assembly (8 total)
├── trackDb.txt # 4 composites
├── {hinge}.multiz.maf.bb # Block 2 — bigMaf
├── chains/{ACC}_to_{ACC_T}.chain.gz # Block 2 — 7 per assembly
├── annot_from_{anchor}.bb # Block 3 — 4 per assembly
├── selection_{strict,relaxed}.bb # Block 4 — PvP01 only
├── orthogroup_membership.bb # Block 4 — PvP01 only
└── Pv4_cohort_on_{ACC}.vcf.gz + .tbi # Block 5 — 1 per assembly
```

Each gene feature in the annotation BigBed tracks gets a `url` attribute in `trackDb.txt` of the form `https://brc-analytics.org/organisms/5855/pangenome/orthogroup/$$` so clicking a gene in the UCSC browser lands on the per-OG panel in BRC-analytics.

## 4-PR sequence

| PR | Adds | Risk |
|----|------|------|
| 1 | Schema additions (LinkML in `catalog/py_package/catalog_build/schema/pangenomes.yaml`). Empty `catalog/source/pangenomes.yml`. `build_pangenomes` step → `catalog/output/pangenomes.json` + `assembly-artifacts.json`. GCA↔GCF translation for Sal-I. | Low — purely additive |
| 2 | Ingest Pv-v1 artifacts from Dropbox → TACC + datacache. `publish_ucsc_hub` step generating per-assembly `trackDb.txt` + `hub.txt` + `genomes.txt`. Concat + tabix-index the cohort VCFs. Rsync to `hgdownload.soe.ucsc.edu/hubs/BRC/`. | Low — static-file generation |
| 3 | "Pangenome" tab on the *P. vivax* organism page (`/organisms/5855`) with the 4 sections above. Orthogroup detail route at `/organisms/5855/pangenome/orthogroup/{id}`. MSAViewer.js + phylotree.js modals. HyPhy-Vision deep-link (pending `veg/hyphy-vision#892`). | Medium — UI |
| 4 | `POPULATION_GENOMICS` + `SELECTION_ANALYSIS` workflow categories. IWC workflow registration for `pangenome-build-and-project` + `hyphy-selection-screen-from-ortholog-table` so users can re-run the pipeline on other panels. | Medium — Galaxy integration |

Mergeable independently. Suggested cadence: PR 1 first, then PR 2 + PR 3 in parallel, PR 4 last.

## What developers need from me

1. **GitHub repo** (public): https://github.com/nekrut/Pv4-pangenome
2. **Dropbox folder** (world-readable): https://www.dropbox.com/scl/fo/gx1mta4adubja4bsxxmgm/AOni3YRX8TS1E-saUblX-eo?rlkey=0ksw8a5hhkxhy3sljqyycr7bu&dl=0
3. **The 6 doc files in `v3/writeup/`** — [`BRC_DEPLOYMENT.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/BRC_DEPLOYMENT.md) is the deployment sketch; the other 5 ([`PANGENOME.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/PANGENOME.md), [`MULTIZ.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MULTIZ.md), [`ORTHOLOGY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/ORTHOLOGY.md), [`MSA_HYPHY.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MSA_HYPHY.md), [`MALARIAGEN_VCF_PROJECTION.md`](https://github.com/nekrut/Pv4-pangenome/blob/main/v3/writeup/MALARIAGEN_VCF_PROJECTION.md)) describe each block in detail with file paths + recreate recipes
4. **MD5 manifest**: `v3/writeup/LARGE_FILES_DROPBOX.tsv` (132 entries)

## What I need from developers

- A staging area on `datacache.galaxyproject.org/brc/data/pangenomes/plasmodium-vivax-v1/` and on TACC to rsync the Dropbox contents into
- A `manifest.json` generator (probably an `aws s3 ls` equivalent on TACC) so `build_pangenomes` can auto-populate track lists
- The 4-PR sequence implemented

## Related issues

- veg/hyphy-vision#892 — CORS + deep-link support for `?json=` queries (gates PR 3's per-OG panel deep-link)
- The old version of this issue body had a more elaborate `PangenomeView` design — that's been replaced by the simpler "tab on organism page" approach reflected here.

## Companion strategic context

Two strategic issues drafted earlier for the `BRC-research` repo describe the broader gap-filling roadmap and the primary-dataset ingest map. The `v3/writeup/BRC_DEPLOYMENT.md` doc in the Pv4-pangenome repo is the implementation slice for the first deliverable in that roadmap.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with v3/writeup/BRC_DEPLOYMENT.md and the four-PR sequence, then inspect the existing organism-page tab pattern from PRs #1261, #1263, #1274, #1277, and #1278. For the initial schema slice, read catalog/py_package/catalog_build/schema/pangenomes.yaml and catalog/source/pangenomes.yml; done requires the catalog outputs and subsequent UI, hosting, and workflow pieces to be implemented as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
build-system, data, devops, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.