promtool: no tool to find inconsistent classic histograms in a TSDB

Proposal

Quantiles, alerts and recording rules can be computed from classic histograms whose stored buckets
contradict each other — and nothing lists which series are affected, or since when.

How the data gets there. No ingest path cross-checks the float series of a classic histogram:
scrape, remote write and the OTLP receiver append what was sent. Buckets whose cumulative counts
decrease, or a le="+Inf" bucket disagreeing with _count, are stored as-is. Native histograms are
rejected at append (model/histogram/histogram.go:474, ErrHistogramCountMismatch); classic ones have
no equivalent check anywhere.

What it costs while it sits there. histogram_quantile repairs non-monotonic input on the fly
(util/annotations/annotations.go:163) and keeps returning plausible numbers, so SLOs and alerts built
on an affected histogram are computed from silently patched data. The only signal is a per-query info
annotation, phrased as a property of the query — the data stays broken for every next query. That
annotation had to be taught to at least name the affected metric (#15411); finding all affected series
in a TSDB still has no tool.

The fix. promtool tsdb check-histograms [db path]
scans a TSDB range (--min-time/--max-time) and reports, per histogram: non-monotonic cumulative
buckets; le="+Inf" disagreeing with _count; le labels that do not parse as a float (le="NaN" is
storable today); duplicate le spellings carrying different values at the same timestamp (le="1" vs
le="1.0" on disjoint timestamps — the v3 migration shape — is recognized as clean); histograms with no
+Inf bucket at all. Exits non-zero when anything is found, so it
can run from cron or CI. Placed under promtool tsdb because it takes a database path like dump and
analyze (promtool check takes config files and server URLs), and marked [Experimental] like
dump-openmetrics.

Limits. Native histograms are out of scope — already validated at append. And I have not measured
how often such data occurs in the wild — this tool is how an operator answers that for their own TSDB.

Found and verified on main at 44d6a0e; file and line references are against that commit.

cc @dgl — promtool is your path, and this proposes a new subcommand in it.

Start with the existing promtool tsdb dump and analyze subcommands, then read model/histogram/histogram.go:474 and util/annotations/annotations.go:163 to understand validation and repair behavior. Use those entry points to map the requested time-range scan and the listed classic-histogram cases, including experimental placement and non-zero exit status.