bigbio / bigbio/hvantk

drift: uniprot_ptm cannot detect a UniProt release; hgnc/gencc have no corroborating signal

Open
#271 0 comments 0 reactions 1 assignee Claimed by @ypriverol View on GitHub
Dominant language
Python
Stars
1
Forks
0
Avg merge
1d 3h
Merged PRs (30d)
40

Description

> **Corrected.** The first version of this issue listed five probes and claimed `expression_atlas`, `msigdb` and `uniprot_ptm` recorded no `source_version`. That was wrong — it came from grepping for `Content-Length` rather than reading what each probe actually returns. `expression_atlas` and `msigdb` are fine and are removed below. The real finding is narrower and sharper: **`uniprot_ptm` is blind to everything except the API's response shape.**

Some drift probes detect a *schema* change but not a *content* change, because the only thing they hash is a column-header or key list. Reading a fingerprint diff where the checksum is unchanged as "the data is unchanged" is therefore wrong — it means "the columns are unchanged."

## What each probe actually carries

| Probe | `source_version` (live value) | Checksum covers | Verdict |
|---|---|---|---|
| `uniprot_ptm` | **`None`** | keys of the *first result record* | **blind to content** |
| `hgnc` | `Last-Modified` | first line only | single unguarded signal |
| `gencc` | `Last-Modified` | header line only | single unguarded signal |
| `expression_atlas` | `"4562 experiments"` | full sorted projection of every experiment + `lastUpdate` | fine |
| `msigdb` | `"2026.1.Mm"` (upstream release) | release version | fine |
| `clingen` | *(deliberately none)* | header line | `Content-Length`, **fails closed** |
| `clinvar` | `Last-Modified` | — | `Content-Length` |
| `gwas_catalog` | `Last-Modified` | ETag + `Content-Length` | ETag + `Content-Length` |

## The actual problem: `uniprot_ptm`

Its fingerprint is:

- `source_version: null` — the UniProt REST endpoint returns no `Last-Modified`, and the probe records the `None` without comment.
- `checksums` — `sha256` over `sorted(results[0].keys())`, i.e. `['entryType', 'extraAttributes', 'features', 'genes', 'primaryAccession', 'sequence', 'uniProtKBCrossReferences']`.

Those keys are the **shape of the UniProt API response**, not the data. They change when UniProt changes its API, which is rare. There is no version, no count, no size, and no hash of any annotation content.

A UniProt release — roughly every 8 weeks, routinely adding and revising PTM annotations — would leave this fingerprint byte-identical and report `clean`. The probe cannot currently detect the event it exists to detect.

## Secondary: `hgnc` and `gencc`

Both hash only the header line and rely on `Last-Modified` as the sole content proxy. That works for a static file server, where replacing a file bumps the mtime — but it is unguarded: `head.headers.get("Last-Modified")` records `None` if the header ever goes missing, and the scheduled bot would commit that as the new baseline.

`clingen` already reasons about precisely this and refuses:

> If either the header row or `Content-Length` is absent the probe raises rather than recording a partial fingerprint, because the scheduled drift bot regenerates a drifted baseline automatically: a single transient omission would otherwise be committed as the new baseline and silently retire content detection for good.

This surfaced while triaging drift PR #266 (`hgnc:lookup`), where only `Last-Modified` moved (31 Jul → 4 Aug) with the column list byte-identical. It was merged on the reasonable assumption that an mtime bump means a new release — but the fingerprint gave no way to corroborate that.

## Proposed change

**`uniprot_ptm`** — needs a real signal, not a bigger header hash. Options, cheapest first: the result count from the same query; `extraAttributes.uniParcId` or a hash over a stable sample of accessions; or UniProt's release identifier if the API exposes one. Anything that moves when annotations change.

**`hgnc`, `gencc`** — record `Content-Length` in `extras` alongside `Last-Modified`. Both already issue a HEAD, so this is free on the wire, and it gives the corroborating signal `clingen` has.

Whether to fail closed should be decided per source rather than uniformly. Bump `PROBE_VERSION` for each probe changed; the comparator already ignores that key when diffing.

## Verification

Two consecutive probes per source, confirming the new signal is stable for unchanged content before it becomes a drift trigger — otherwise it reintroduces the "drifted on every run" failure `clingen` hit with `Last-Modified` and documented.

Follow-up from the 0.3.0 release. Not urgent — nothing is broken and no artifact is wrong — but `uniprot_ptm`'s detection is currently inoperative rather than merely weak.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.