developmentseed / developmentseed/montandon-website

Generate the Data Sources page from the published sources.json

Open
#18 0 comments 0 reactions 1 assignee Claimed by @wrynearson View on GitHub
Dominant language
No language data
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Context

`monty-stac-extension` now publishes a machine-readable source index, generated from a `sources.yml` manifest that is the single source of truth for the Monty sources:

**** (live, 15 entries)

```json
{
"sources": [
{
"id": "cems",
"name": "Copernicus Emergency Management Service — Rapid Mapping",
"org": "Copernicus Emergency Management Service (CEMS)",
"url": "https://mapping.emergency.copernicus.eu/",
"license": "Copernicus data policy — free & open (attribution required)",
"status": "etl",
"types": ["event", "hazard", "impact", "response"],
"collections": ["cems-events", "cems-hazards", "cems-impacts", "cems-response"],
"doc": "https://ifrcgo.org/monty-stac-extension/model/sources/CEMS/",
"etl": "https://github.com/IFRCGo/pystac-monty/blob/main/pystac_monty/sources/cems.py"
}
]
}
```

CI on that repo (`scripts/gen_sources_index.py --check`) fails the build if the manifest drifts from the docs nav, from the shipped `examples/` collections, or from the generated index — so `sources.json` is enforced to stay true, not maintained by convention.

`docs/methodology/data-sources.md` here is a hand-maintained parallel list of the same sources. It has already drifted.

## The drift, concretely

Comparing the `Contributes` column on `data-sources.md` against `types` in the published `sources.json` — **6 of the 13 listed sources are wrong today**:

| Source | Page says | Actually contributes | Missing |
|---|---|---|---|
| GDACS | event, hazard | event, hazard, impact | `impact` |
| IDMC | event | event, impact | `impact` |
| IDU | event | event, impact | `impact` |
| IFRC-DREF | event, impact | event, hazard, impact | `hazard` |
| USGS | event, hazard | event, hazard, impact | `impact` |
| Copernicus EMS | response | event, hazard, impact, response | `event`, `hazard`, `impact` |

Also stale:

- **Charter and Copernicus EMS are still marked *(upcoming)***. Both now have transformers in `pystac-monty` ([`charter.py`](https://github.com/IFRCGo/pystac-monty/blob/main/pystac_monty/sources/charter.py), [`cems.py`](https://github.com/IFRCGo/pystac-monty/blob/main/pystac_monty/sources/cems.py)) and are `status: etl` in the manifest.
- **GFD license is "Unknown"** on the page → it is `CC BY-NC-ND 4.0`.
- **PDC license is "Unknown"** on the page → restricted to disaster/emergency-management use, no redistribution without authorization.

None of this is anyone's fault — it's the expected outcome of maintaining the same list in two repos by hand.

## Proposal

Generate the **Overview table** on `docs/methodology/data-sources.md` from `sources.json` at build time, using a [VitePress data loader](https://vitepress.dev/guide/data-loading). Sketch:

```ts
// docs/methodology/sources.data.ts
const URL = "https://ifrcgo.org/monty-stac-extension/sources.json";

export default {
async load() {
const { sources } = await (await fetch(URL)).json();
// `undocumented` = has example collections but no source doc yet;
// internal bookkeeping, not for the public page.
return sources.filter((s) => s.status !== "undocumented");
},
};
```

Two things that make this land cleanly:

- Filtering `status !== "undocumented"` yields **exactly the 13 sources the page already lists** — `alerthub` and `reference` are the only excluded entries, and neither belongs on a public page.
- It's a **build-time** fetch, so there's no client-side request, no CORS surface, and no runtime dependency on `ifrcgo.org`.

`status` could also drive an honest badge, replacing the manual *(upcoming)* marker:

| `status` | meaning | suggested badge |
|---|---|---|
| `production` | orchestrated in [`montandon-etl`](https://github.com/IFRCGo/montandon-etl) | *(none — the default)* |
| `etl` | transformer exists in `pystac-monty`, not yet wired into the ETL | *Integration in progress* |
| `templates` / `analysis` | mapping specified / researched only | *Planned* |

## Open questions

**1. Three fields the page uses aren't in `sources.json`.** The manifest carries `id`, `name`, `org`, `url`, `license`, `status`, `types`, `collections`, `doc`, `etl` — but the page also has a **Type** column (*"International Organization"*, *"Academic / Research"*, …), a **Contact** email per source, and a prose description. So either:

- **(a)** scope the generated part to the Overview table minus the Type column, and keep the prose sections hand-written; or
- **(b)** extend `sources.yml` upstream with `org_type`, `contact`, and a one-line `description`, and generate more of the page.

I'd lean **(b)** for `org_type` + `contact` (they're stable facts that belong next to the rest of the source metadata, and IFRCGo/monty-stac-extension#65 already built the CI to keep them honest), and keep the long prose here — it's editorial, and it's the part this site should own. Happy to send that upstream PR if you agree.

**2. Rebuild trigger.** This site deploys on push to `main` / `workflow_dispatch` only, so a build-time fetch is a snapshot taken at the last deploy. To pick up manifest changes, either add a `schedule:` to `deploy.yml`, or have `monty-stac-extension` fire a `repository_dispatch` here after a docs deploy. The latter is tighter; it needs a token with access to this repo.

**3. Build-time network dependency.** The build starts failing if `ifrcgo.org` is down at that moment. Worth deciding whether to vendor a committed fallback copy or just accept it.

## Refs

- IFRCGo/monty-stac-extension#65 — the `sources.yml` manifest + generated indexes + published `sources.json` (merged: IFRCGo/monty-stac-extension#78)
- IFRCGo/monty-stac-extension#71 — the tracking issue for this consumption, on the extension side
- IFRCGo/monty-stac-extension#62 — parent epic, whose goal 5 is *"a machine-readable source index that `montandon-website` consumes instead of hand-maintaining a duplicate list"*
- Supersedes the hand-maintained list added in #4

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.