Protocol: cluster and display a signature similarity matrix

Tier: A (fieldwork) · Type: atomic · Category: Statistical Analysis

What

Take any signature-by-signature similarity matrix, convert it to a distance, cluster hierarchically, and produce the annotated heatmap and dendrogram — without caring how the similarity was computed.

Why it matters

This exists because of a question raised in review: A5 defined Jaccard similarity at a single rank, C1 defines semantic similarity across mixed ranks, and both are followed by exactly the same clustering and display steps. Bundling those steps into each similarity protocol would duplicate them and, worse, would make "do the A5 analysis but with semantic similarity" impossible to express as a protocol composition.

Splitting the pipe at the similarity matrix makes the measure a swappable input. Anyone can then add a third measure later — Sørensen-Dice, overlap coefficient, a phylogenetic distance — and get the clustering for free.

Source material

• waldronlab/bugSigSimple — vignettes/goldstandard_vignette_peace.Rmd ("Heatmap", "Dendrogram", "Convert similarity to distance", the top/left annotation blocks and signature-length labelling), vignettes/capstoneanalysis_clare.rmd ("Cluster analysis")
• waldronlab/BugSigDBPaper — vignettes/Figure2.Rmd ("Hierarchical clustering on the semantic similarity matrix", "Heatmap comparison" of the two measures, k-means as an alternative)

Scope

In: the required input matrix layout (so any similarity protocol can feed it); similarity-to-distance conversion and what it assumes; linkage method and why the default is not left to the implementation; how many clusters and how that is chosen; annotation of rows/columns by study, condition, body site and signature length; what the dendrogram may and may not be read as.

Out: computing similarity; testing clusters for condition enrichment (that is a Tier C protocol).

Frontmatter starting point

type: "atomic"
category: "Statistical Analysis"
citation: ""   # needs the original source of the clustering method actually specified —
               # e.g. Ward 1963 (10.1080/01621459.1963.10500845) if Ward linkage. VERIFY.
tags: [clustering, hierarchical-clustering, heatmap, dendrogram, similarity-matrix]

The citation must name whoever proposed the clustering method the protocol specifies, not the BugSigDB paper. If the protocol allows several linkages, see the discussion of alternatives in CONTRIBUTING-adjacent issues — but note that permitting several linkages from several different papers conflicts with the one-citation rule, so the protocol should specify one and justify it.

Acceptance criteria

• Input matrix layout is specified independently of any particular similarity measure
• Demonstrably works when fed either the Jaccard or the semantic similarity matrix
• Linkage is specified and justified, not deferred to an implementation default
• States plainly what a dendrogram branch does and does not imply

Cite the method's origin, not its users

PROTOCOL_STANDARD.md is explicit: an atomic protocol carries "strictly 1 citation... corresponding
to the primary literature where the method was originally published." Find the paper that proposed
the method. Do not cite a paper that merely applied it — including the BugSigDB and curatedMetagenomicData
papers, which are the source of the analysis these protocols were extracted from but almost never the
source of the method.

Tracing a method back to its first publication is real work, and it is part of the task, not a
formality. Three things to expect:

• Candidate DOIs in this issue are leads, not answers. Anything marked VERIFY has not been checked.
• Some methods predate modern citation practice or have no single identifiable origin. If that is
  genuinely the case, say so in the pull request rather than reaching for a convenient recent paper.
  Raise it as an issue in waldronlab/agent-protocol-standard — the standard may need a way to express
  "classical method, no primary source".
• If you cannot name one paper that proposed everything the protocol does, it is more than one
  protocol. That test has now split four protocols out of this batch: enrichment into three methods,
  filtering from transformation, LODO from random forest, and PERMANOVA from ANOSIM.

Where the lab's own paper genuinely did propose the method — the oral-to-gut score, and LODO
cross-validation in Pasolli et al. 2016 — citing it is correct. That is the exception, not the pattern.

Before you start

Read CONTRIBUTING.md and
PROTOCOL_STANDARD.md.
The format is defined in the standard repo, not this one. Protocols are prose, not code: they say what to do
and why, precisely enough that two people — or two agents, in two languages — get the same answer. The existing
independent-filtering-variance
protocol is the model to imitate for tone and level of detail.

Validate locally before opening the PR:

git clone https://github.com/waldronlab/agent-protocol-standard.git
Rscript agent-protocol-standard/scripts/validate-protocol.R protocols

Read CONTRIBUTING.md and PROTOCOL_STANDARD.md first, then use protocols/independent-filtering-variance/protocol.md as the format model. Review the cited BugSigSimple and BugSigDBPaper vignette sections, specify the matrix layout, linkage, cluster choice, annotations, and dendrogram interpretation, and verify the method's primary citation. Validate the finished protocol with Rscript agent-protocol-standard/scripts/validate-protocol.R protocols.