Composite protocol: benchmark microbe set enrichment methods against each other

Tier: C (BugSigDB paper) · Type: composite · Category: Statistical Analysis

What

Run ORA, PADOG and CBEA over the same data and the same BugSigDB microbe sets, and compare what they find — the analysis Figure3.Rmd performs, expressed as a composition of the three method protocols.

Why it matters

This is where the "which method should I use?" question belongs. Putting the choice inside a single protocol would have hidden the fact that the three methods disagree — which is precisely why the paper benchmarked them. A composite makes the comparison itself the deliverable and keeps each method's assumptions stated in its own protocol.

It also gives the repository a worked example of the pattern for "several methods, one question", which will recur.

Source material

• waldronlab/BugSigDBPaper — vignettes/Figure3.Rmd in full, including the differential abundance input, the ORA/PADOG comparison, and CBEA run both pooled and per-dataset
• Paper: 10.1038/s41587-023-01872-y

Scope

In: the shared input preparation and how identical inputs are guaranteed across methods; the concordance measures used to compare results; how disagreement is reported; guidance on which method suits which situation, grounded in the comparison rather than asserted.

Out: every method's own steps.

Frontmatter starting point

type: "composite"
category: "Statistical Analysis"
citation: "10.1038/s41587-023-01872-y"   # legitimate here: the composite pipeline is the paper's
protocols_used:
  - name: "ora-microbe-set-enrichment"
  - name: "padog-microbe-set-enrichment"
  - name: "cbea-microbe-set-enrichment"
  - name: "differential-abundance-input"   # or whichever protocol supplies the ranking

Acceptance criteria

• All three methods receive provably identical inputs
• Concordance and disagreement are both reported, with a defined measure
• Blocked on C4, C5 and C6 merging first

Cite the method's origin, not its users

PROTOCOL_STANDARD.md is explicit: an atomic protocol carries "strictly 1 citation... corresponding
to the primary literature where the method was originally published." Find the paper that proposed
the method. Do not cite a paper that merely applied it — including the BugSigDB and curatedMetagenomicData
papers, which are the source of the analysis these protocols were extracted from but almost never the
source of the method.

Tracing a method back to its first publication is real work, and it is part of the task, not a
formality. Three things to expect:

• Candidate DOIs in this issue are leads, not answers. Anything marked VERIFY has not been checked.
• Some methods predate modern citation practice or have no single identifiable origin. If that is
  genuinely the case, say so in the pull request rather than reaching for a convenient recent paper.
  Raise it as an issue in waldronlab/agent-protocol-standard — the standard may need a way to express
  "classical method, no primary source".
• If you cannot name one paper that proposed everything the protocol does, it is more than one
  protocol. That test has now split four protocols out of this batch: enrichment into three methods,
  filtering from transformation, LODO from random forest, and PERMANOVA from ANOSIM.

Where the lab's own paper genuinely did propose the method — the oral-to-gut score, and LODO
cross-validation in Pasolli et al. 2016 — citing it is correct. That is the exception, not the pattern.

Before you start

Read CONTRIBUTING.md and
PROTOCOL_STANDARD.md.
The format is defined in the standard repo, not this one. Protocols are prose, not code: they say what to do
and why, precisely enough that two people — or two agents, in two languages — get the same answer. The existing
independent-filtering-variance
protocol is the model to imitate for tone and level of detail.

Validate locally before opening the PR:

git clone https://github.com/waldronlab/agent-protocol-standard.git
Rscript agent-protocol-standard/scripts/validate-protocol.R protocols

Read CONTRIBUTING.md and PROTOCOL_STANDARD.md, then inspect vignettes/Figure3.Rmd and the independent-filtering-variance protocol for the expected prose format. Confirm the shared input, concordance measure, disagreement reporting, and primary citations for ORA, PADOG, and CBEA. Validate with Rscript agent-protocol-standard/scripts/validate-protocol.R protocols after C4, C5, and C6 are merged.