Protocol: resampling test for signature replicability within a condition

Tier: C (BugSigDB paper) · Type: atomic · Category: Statistical Analysis

What

Test whether signatures from independent studies of the same condition are more similar to each other than expected by chance, by comparing observed within-condition similarity to a null built from randomly sampled signatures drawn from the same body site, using a one-sided resampling test.

Why it matters

"Do independent studies of the same disease find the same microbes?" is the question BugSigDB exists to answer, and this is the test that answers it. It is also the natural upgrade path from the Tier A Monte Carlo protocol — same logic, similarity instead of frequency — which makes it a good second protocol for a student who has already written that one.

Source material

• waldronlab/BugSigDBPaper — vignettes/Figure2.Rmd ("Test replicability across body sites", "Test each condition for semantic similarity")
• Paper: 10.1038/s41587-023-01872-y (one-sided resampling across 311 human fecal studies)

Scope

In: the resampling universe and why it is body-site-matched; whether signatures from the same study may be paired (they must not); the summary statistic for within-condition similarity; number of resamples; one-sided direction; multiple testing across conditions; the minimum number of independent studies for a condition to be testable.

Out: the similarity measure itself.

Frontmatter starting point

type: "atomic"
category: "Statistical Analysis"
citation: ""   # A permutation/resampling test — Fisher 1935 or Pitman 1937. VERIFY. The BugSigDB
               # paper applied it to signatures; it did not propose resampling.
protocols_used:
  - name: "taxonomic-semantic-similarity"
tags: [bugsigdb, replicability, resampling, permutation, meta-research]

Acceptance criteria

• Same-study pairs are explicitly excluded, with the reason stated
• The testable-condition threshold is a stated parameter
• Multiple-testing adjustment across conditions is specified

Cite the method's origin, not its users

PROTOCOL_STANDARD.md is explicit: an atomic protocol carries "strictly 1 citation... corresponding
to the primary literature where the method was originally published." Find the paper that proposed
the method. Do not cite a paper that merely applied it — including the BugSigDB and curatedMetagenomicData
papers, which are the source of the analysis these protocols were extracted from but almost never the
source of the method.

Tracing a method back to its first publication is real work, and it is part of the task, not a
formality. Three things to expect:

• Candidate DOIs in this issue are leads, not answers. Anything marked VERIFY has not been checked.
• Some methods predate modern citation practice or have no single identifiable origin. If that is
  genuinely the case, say so in the pull request rather than reaching for a convenient recent paper.
  Raise it as an issue in waldronlab/agent-protocol-standard — the standard may need a way to express
  "classical method, no primary source".
• If you cannot name one paper that proposed everything the protocol does, it is more than one
  protocol. That test has now split four protocols out of this batch: enrichment into three methods,
  filtering from transformation, LODO from random forest, and PERMANOVA from ANOSIM.

Where the lab's own paper genuinely did propose the method — the oral-to-gut score, and LODO
cross-validation in Pasolli et al. 2016 — citing it is correct. That is the exception, not the pattern.

Before you start

Read CONTRIBUTING.md and
PROTOCOL_STANDARD.md.
The format is defined in the standard repo, not this one. Protocols are prose, not code: they say what to do
and why, precisely enough that two people — or two agents, in two languages — get the same answer. The existing
independent-filtering-variance
protocol is the model to imitate for tone and level of detail.

Validate locally before opening the PR:

git clone https://github.com/waldronlab/agent-protocol-standard.git
Rscript agent-protocol-standard/scripts/validate-protocol.R protocols

Read CONTRIBUTING.md, PROTOCOL_STANDARD.md, and the independent-filtering-variance protocol for required format and level of detail. Use the BugSigDBPaper references as analysis leads, then verify the method’s primary citation and document the body-site-matched resampling universe, exclusions, threshold, statistic, resamples, direction, and multiple-testing adjustment. Validate with the provided Rscript command; done means all acceptance criteria are covered and validation passes.