Migrate direct raw-S3 bbop-sqlite consumers to the semanticsql CDN

@caufieldjh is already working on this.

Since Jul 14, 2026.

[bot] Opened by a Claude Code agent on behalf of @kltm. **Body updated 2026-08-18 (2nd revision):** every entry below is now verified against **fetched default-branch file content** (literal-match), not code-search hits — search tokenizes on hyphens and produced several false positives, now removed. Verdicts also cross-checked against raw-bucket access logs (2026-07-08 → 2026-08-18) and per-file git history.

Companion to #112 (remove raw-bucket access). These projects hardcode the raw bucket URL directly, so they will **not** auto-migrate via oaklib/ODK bumps; they need to repoint to `https://semanticsql.berkeleybop.io` (drop-in, same paths) before raw access is removed.

**Gotcha RETIRED 2026-08-18:** the CDN no longer 403s `Python-urllib/*` User-Agents (host-scoped Browser Integrity Check exemption, verified). All client types work; the swap is a plain one-line change.

## Verified still-present (literal raw URL in current default-branch runtime code)

- [ ] monarch-initiative/monarch-ingest — `src/monarch_ingest/download.yaml` L28 — outreach monarch-initiative/monarch-ingest#711 (open, no engagement) — phenio pulls also confirmed in access logs
- [ ] Knowledge-Graph-Hub/kg-alzheimers — `src/kg_alzheimers/download.yaml` L186/L255 — outreach Knowledge-Graph-Hub/kg-alzheimers#78 (open, no engagement) — traffic-confirmed
- [ ] monarch-initiative/dictybase-ingest — `download.yaml` L10; ⚠️ `src/versions.py` filters on `contains=["bbop-sqlite"]`, so the URL fix must also update that selector or version-tracking silently breaks — outreach monarch-initiative/dictybase-ingest#3 (open, no engagement) — traffic-confirmed
- [ ] biobricks-ai/semsql — `stages/01_download.sh` (URL generator); ⚠️ `stages/00_invalidate.sh` does anonymous `aws s3 ls` (**bucket-listing dependency**, see below); `.bb/source.jsonld` metadata — outreach biobricks-ai/semsql#1 (open, no engagement)
- [ ] Knowledge-Graph-Hub/universalizer — `universalizer/oak_utils.py` L48 (note: fetches **uncompressed `.db`** objects) — outreach Knowledge-Graph-Hub/universalizer#41 (open, no engagement)
- [ ] microbiomedata/ontology-loader — `src/ontology_loader/ontology_processor.py` L100 — outreach microbiomedata/ontology-loader#59 (open, no engagement). Covers microbiomedata/nmdc-runtime too (its notebook just calls this library).
- [ ] ccb-hms/NHANES-metadata — `code/generate_ontology_tables.py` L30 — outreach ccb-hms/NHANES-metadata#37 (open, no engagement)
- [ ] waldronlab/bioanalyzer-backend — `app/normalization/grounding/seed.py` L57 (`SEMANTIC_SQL_BASE_URL`) — outreach waldronlab/bioanalyzer-backend#124
- [ ] seandavi/metacurator — `src/metacurator/grounding/local_duckdb.py` L26 (single base-URL constant; cleanest possible migration) — outreach seandavi/metacurator#29 — hancestro pulls confirmed in access logs
- [ ] seandavi/cdsci-lake — `src/cdsci/lake/config.py` L141 + `sources/ontology/ingest.py`; ⚠️ its registry IS the bucket listing (paginated ListObjectsV2) — outreach via comment on seandavi/cdsci-lake#40
- [ ] CultureBotAI/MediaIngredientMech — `scripts/check_chebi_currency.py` L53 — outreach via comment on CultureBotAI/MediaIngredientMech#206
- [ ] microbiomedata/external-metadata-awareness — `notebooks/environmental_context_value_sets/generate_voting_sheet.ipynb` (source cell) + ⚠️ `notebooks/multi-lexmatch/interleave_s3_catalog_yaml_registry_bioportal_obo.ipynb` calls the S3 **ListBucket XML API**; 2 docs files — outreach microbiomedata/external-metadata-awareness#553
- [ ] Knowledge-Graph-Hub/kg-registry — `resource/reactome/reactome*.md` `product_url:` frontmatter — **machine-consumable registry metadata**, not prose; directs downstream users to the raw URL — outreach Knowledge-Graph-Hub/kg-registry#701

## ⚠️ Bucket-LISTING dependents (URL swap alone does not migrate these)

Four verified consumers enumerate the bucket rather than (only) fetching objects: **ontoProc2** (post-migration code parses ListBucket XML from the CDN root), **cdsci-lake** (ListObjectsV2 registry), **biobricks `00_invalidate.sh`** (`aws s3 ls`), **external-metadata-awareness** (ListBucket XML notebook). Live probes 2026-08-18: the CDN root currently proxies the bucket's V1 listing (works only while the bucket stays public), and CloudFront strips query strings, so V2/pagination params are silently ignored (fine at ~332 keys; breaks at 1,000). **#112 must decide a listing strategy**: grant `s3:ListBucket` to the CDN origin access at lockdown, or publish a manifest file and migrate these four to it.

## Migrated (verified in current code — done or nearly)

- [x] Knowledge-Graph-Hub/kg-microbe — repointed to the CDN 2026-07-21 (`9c8ddcad`, via #595). Residual raw-log traffic through 2026-08-18 attributed to stale checkouts/deployments, not master.
- [x] vjcitn/ontoProc2 — runtime repointed 2026-07-29 (`1d4df64d`, via vjcitn/ontoProc2#11, still open): remaining items are the README `aws s3 ls s3://bbop-sqlite/` example and the listing caveat above. vjcitn/op2workshop README carries the same example.
- [x] monarch-initiative/dismech, ai4curation/ai-gene-review — oaklib lock bumps landed 2026-08-07 / 2026-08-12; verified in traffic.

## Docs/examples only (verified literals, prose/log context; fix opportunistically)

- [ ] cthoyt/pystow — docstring examples (`api.py`, `impl.py`)
- [ ] berkeleybop/metpo — 3 docs files (its script uses `sqlite:obo:` selectors — migrates with oaklib); turbomam/metpo-attic — 4 docs files incl. a copy-pasteable `curl` example
- [ ] brad-usredoxlabs/computable-lab — 1 doc
- [ ] monarch-initiative/ontogpt-experiments — committed stdout log of an old run (evidence, not config)

## Removed after verification (false positives of hyphen-tokenized code search)

monarch-initiative/rare-disease-identification (docstring prose only; runtime uses a local OBO), monarch-initiative/mondo notebook (URL only in saved output cells; source uses `sqlite:obo:` selectors), several name-only doc mentions elsewhere. Selector-based (`sqlite:obo:`) consumers are out of scope here — they migrate via oaklib ≥0.7.2, with the standing caveat that **pinned lockfiles do not auto-migrate** (three CI consumers to date required manual lock bumps).

## Coordination notes

- **Outreach-by-issue has had near-zero engagement**: of the 2026-07-14 wave, only kg-microbe (silent fix) and ontoProc2 (same-day author response) acted; the other six issues have zero comments after five weeks. Minimal PRs are likely the higher-yield instrument for the remaining repos.
- **#114 remains the blocker** for `semsql download` / library-default users: semsql on PyPI is still 0.4.0 (2025-02-05).
- **The other ecosystem-wide unlock is an ODK release**: the bundled-oaklib bump (INCATools/ontology-development-kit#1354) is merged but unreleased — the newest stable image (`odkfull:v1.6.1`) still bundles oaklib 0.6.23, so every ODK-based ontology repo's CI pulls from the raw bucket and **cannot migrate by its own action** until ODK ships (verified 2026-08-18: only `odkfull:dev` carries the fix). Release request filed: INCATools/ontology-development-kit#1368.
- Populations with no discoverable repo (cloud notebooks, one-shot users, private code) remain reachable only via release notes / announcements before #112.

_— Posted by Claude Code agent on behalf of @kltm._