galaxyproject / galaxyproject/brc-analytics

Add BRC datasets to Google Datasets catalog

Open
#1,264 0 comments 0 reactions 1 assignee Claimed by @frano-m View on GitHub
roadmap item
Dominant language
TypeScript
Stars
7
Forks
11
Avg merge
2d 12h
Merged PRs (30d)
16

Description

Add BRC datasets to the [Google Dataset Search](https://datasetsearch.research.google.com/) catalog so assemblies are discoverable from Google. This is done by embedding [schema.org Dataset](https://schema.org/Dataset) JSON-LD on the relevant pages — Google's crawler picks it up.

## Reference implementation

NCPI Dataset Catalog has already shipped this for studies. Mirror their pattern:

- Builder + types: [`app/utils/schemaOrg.ts`](https://github.com/NIH-NCPI/ncpi-dataset-catalog/blob/main/app/utils/schemaOrg.ts) — `SchemaDataset` interface and `buildStudyJsonLd()` factory.
- Render component: [`app/components/Detail/components/StudyJsonLd/studyJsonLd.tsx`](https://github.com/NIH-NCPI/ncpi-dataset-catalog/blob/main/app/components/Detail/components/StudyJsonLd/studyJsonLd.tsx) — wraps the JSON-LD in a `` inside `next/head`, with HTML escaping to prevent script injection.
- Page integration: [`pages/[entityListType]/[...params].tsx`](https://github.com/NIH-NCPI/ncpi-dataset-catalog/blob/main/pages/%5BentityListType%5D/%5B...params%5D.tsx) — mounted on the detail route only.
- Tests: [`app/utils/schemaOrg.test.ts`](https://github.com/NIH-NCPI/ncpi-dataset-catalog/blob/main/app/utils/schemaOrg.test.ts) — covers required fields, truncation, and conditional fields.

## Google Dataset required + recommended fields

Per [Google's Dataset structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/dataset):

**Required**
- `name` — descriptive title
- `description` — 50–5000 characters

**Recommended**
- `identifier`, `url`, `sameAs`
- `creator`, `funder`, `license`
- `distribution` (with `contentUrl`, `encodingFormat`)
- `keywords`, `variableMeasured`, `measurementTechnique`
- `spatialCoverage`, `temporalCoverage`
- `includedInDataCatalog`, `isAccessibleForFree`, `version`, `citation`

## Initial mapping — assembly (`BRCDataCatalogGenome`) → `Dataset`

Source entity at [`app/apis/catalog/brc-analytics-catalog/common/entities.ts`](app/apis/catalog/brc-analytics-catalog/common/entities.ts) (`BRCDataCatalogGenome`).

| schema.org field | Source / value |
| --- | --- |
| `@context` | `\"https://schema.org\"` |
| `@type` | `\"Dataset\"` |
| `name` | `${taxonomicLevelSpecies}${strainName ? \" — \" + strainName : \"\"} (${accession})` |
| `description` | Composed sentence: species + strain + accession + assembly level + chromosome/scaffold counts + length. Strip HTML, truncate to 5000 chars. |
| `identifier` | `[accession, ncbiTaxonomyId]` (NCBI assembly accession + taxonomy ID) |
| `url` | `${browserURL}/data/organisms/${ncbiTaxonomyId}/genomes/${accession}` (or whichever route resolves the assembly detail page) |
| `sameAs` | NCBI assembly URL: `https://www.ncbi.nlm.nih.gov/datasets/genome/${accession}/` |
| `includedInDataCatalog` | `{ \"@type\": \"DataCatalog\", name: \"BRC Analytics\", url: browserURL }` |
| `isAccessibleForFree` | `true` |
| `keywords` | `[priorityPathogenName, ...taxonomicGroup, taxonomicLevelKingdom, taxonomicLevelFamily, taxonomicLevelGenus, taxonomicLevelSpecies].filter(Boolean)` |
| `version` | `accession` version suffix (e.g. trailing `.1`/`.2`) if parseable |
| `distribution` | Build `DataDownload[]` from any of: `galaxyDatacacheUrl`, `geneModelUrl`, `ucscBrowserUrl` (when present). Set `encodingFormat` per source (`application/gzip`, `text/x-gff3`, `text/html`). |
| `variableMeasured` | Optional: `[\"chromosomes\", \"scaffoldCount\", \"scaffoldN50\", \"scaffoldL50\", \"gcPercent\", \"length\", \"coverage\"]` filtered to populated values, encoded as `PropertyValue` objects. |
| `creator` | Optional: `{ \"@type\": \"Organization\", name: \"BRC Analytics\" }` until per-assembly source attribution is available. |
| `license` | TBD — confirm with team (likely a permissive public-domain notice for the metadata). |

Open questions for `variableMeasured` / `creator` / `license` should be resolved before merge.

## Implementation steps

1. Add `app/utils/schemaOrg.ts` with `SchemaDataset` types and `buildAssemblyJsonLd(genome, browserURL)`.
2. Add `app/components/.../AssemblyJsonLd` component that renders the JSON-LD via `next/head` with the same HTML-escape helper as NCPI.
3. Mount the component on the assembly detail page (`pages/data/[entityListType]/[entityId]/index.tsx`).
4. Unit-test the builder: required fields present, description truncation, conditional fields omitted when source is null.
5. Validate output against [Google's Rich Results Test](https://search.google.com/test/rich-results) and [Schema Markup Validator](https://validator.schema.org/) for a few representative assemblies (high-priority pathogen, eukaryote, virus).
6. Once shipped, request indexing via Google Search Console and confirm dataset pages start appearing in Google Dataset Search.

## Out of scope (follow-ups)

- Mapping for organism pages (`BRCDataCatalogOrganism`).
- Sitemap entries for assembly detail pages if not already present.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.