Protocol: taxon frequency table with a binomial test of reported direction

Tier: A (fieldwork) · Type: atomic · Category: Statistical Analysis

What

For each taxon appearing in a set of signatures, count how many signatures report it increased in cases versus decreased, and test H0: a taxon is equally likely to be reported increased or decreased (binomial, p = 0.5).

Why it matters

This is the workhorse result of the fieldwork analyses, and it is also the one most often misread. It also needs to state what counts as an independent report (signature? experiment? study?), which is the real methodological question here and is currently answered differently in different vignettes.

The power limitation is a required part of the protocol, not a footnote

The protocol must state this before the test is described, because it determines whether running the test is worthwhile at all:

• The two-sided binomial test at p = 0.5 cannot reach significance below 6 independent reports, no matter how consistent the direction. With perfect concordance across 5 studies, the smallest attainable p-value is 2 × 0.5⁵ = 0.0625. With 6, it is 2 × 0.5⁶ = 0.03125 — the first count that clears 0.05.
• 6 is therefore the point at which the test becomes capable of a significant result, not the point at which it becomes useful. At 6 studies, only perfect concordance can be detected; a single discordant report drops it back below significance.
• At least 10 independent reports is the appropriate working minimum. At 10, the test tolerates one discordant report (p = 0.021) and still detects the effect, so the result reflects the direction of the evidence rather than the absence of any exception.

A taxon below the minimum is reported with its counts and no p-value, rather than with a non-significant one — an underpowered null result invites exactly the wrong reading.

Source material

• waldronlab/bugSigSimple — R/describe_curation.R (createTaxonTable, .countTaxon, .createBinomTestSummary), R/simple.R (getMostFrequentTaxa)
• vignettes/capstoneanalysis_fatima.rmd — the manual binomial test this generalizes
• vignettes/capstoneanalysis_clare.rmd — "Overall frequencies of taxa increased/decreased in cases"

Scope

In: the unit of independent observation; the minimum-report thresholds above and how under-powered taxa are reported; counting rules for a taxon reported at multiple taxonomic levels; the binomial test and its minimum-n limitation; multiple-testing treatment across taxa; tie and zero-count handling.

Out: the Monte Carlo recurrence null (separate protocol); plotting.

Frontmatter starting point

type: "atomic"
category: "Statistical Analysis"
citation: "10.1098/rstl.1710.0011"   # Arbuthnott 1710 — the sign test, plausibly the first
                                     # published significance test. VERIFY this is the right
                                     # primary source for the binomial test as used here.
tags: [bugsigdb, binomial-test, differential-abundance, direction, frequency]

Acceptance criteria

• States both thresholds with the arithmetic: 6 to be capable of significance, ≥ 10 as the working minimum
• Defines how a taxon below the minimum is reported (counts, no p-value)
• Defines the independence unit explicitly and requires it to be recorded
• Specifies the multiple-testing adjustment (or explicitly declines one, with justification)

Cite the method's origin, not its users

PROTOCOL_STANDARD.md is explicit: an atomic protocol carries "strictly 1 citation... corresponding
to the primary literature where the method was originally published." Find the paper that proposed
the method. Do not cite a paper that merely applied it — including the BugSigDB and curatedMetagenomicData
papers, which are the source of the analysis these protocols were extracted from but almost never the
source of the method.

Tracing a method back to its first publication is real work, and it is part of the task, not a
formality. Three things to expect:

• Candidate DOIs in this issue are leads, not answers. Anything marked VERIFY has not been checked.
• Some methods predate modern citation practice or have no single identifiable origin. If that is
  genuinely the case, say so in the pull request rather than reaching for a convenient recent paper.
  Raise it as an issue in waldronlab/agent-protocol-standard — the standard may need a way to express
  "classical method, no primary source".
• If you cannot name one paper that proposed everything the protocol does, it is more than one
  protocol. That test has now split four protocols out of this batch: enrichment into three methods,
  filtering from transformation, LODO from random forest, and PERMANOVA from ANOSIM.

Where the lab's own paper genuinely did propose the method — the oral-to-gut score, and LODO
cross-validation in Pasolli et al. 2016 — citing it is correct. That is the exception, not the pattern.

Before you start

Read CONTRIBUTING.md and
PROTOCOL_STANDARD.md.
The format is defined in the standard repo, not this one. Protocols are prose, not code: they say what to do
and why, precisely enough that two people — or two agents, in two languages — get the same answer. The existing
independent-filtering-variance
protocol is the model to imitate for tone and level of detail.

Validate locally before opening the PR:

git clone https://github.com/waldronlab/agent-protocol-standard.git
Rscript agent-protocol-standard/scripts/validate-protocol.R protocols

Read CONTRIBUTING.md and PROTOCOL_STANDARD.md first, then use independent-filtering-variance/protocol.md as the prose model. Compare R/describe_curation.R, R/simple.R, and the two cited vignettes while resolving the independence, threshold, counting, and citation questions. Draft the protocol with all acceptance criteria, then run the repository validator command before opening a PR.