Benchmark methodology for the published MOSS-TTS-Realtime RTF / TTFB numbers

We are evaluating MOSS-TTS-Realtime for a telephony voice agent and cannot
reproduce the published figures. We found that metric definition materially
changes the reported result. Because the published setup does not include the
fixture or benchmark script, we cannot determine how much of the remaining
difference comes from hardware, fixture, warmup coverage, or aggregation.

What the model card states

The model achieves a TTFB of 180ms (after warmup) with an RTF of 0.51 on a
single L20 GPU.

The commit that introduced these numbers does not include the benchmark script,
and the card does not state the fixture, sample count, or aggregation, so we
could not replicate the setup exactly.

What we measured

Your reference Transformers implementation, not a third-party serving stack:

Results, computed two ways over the same 100 responses:

The gap between the two rows is trailing and leading silence in the returned
audio. Leading silence alone had a p95 of 3.675 s.

Speech-span detection, so the second row is reproducible: 20 ms RMS frames, a
speech threshold of max(-60 dBFS, 2% of peak frame RMS), only leading and
trailing silence trimmed, internal pauses preserved.

Why we are asking

Reading your serving code, RTF appears to be computed against the full returned
audio duration. That is a perfectly reasonable throughput metric and we are not
suggesting it is wrong.

Because the hardware, fixture, and benchmark boundaries differ, our 0.896
neither reproduces nor contradicts 0.51. It only demonstrates that the metric
definition materially changes the conclusion on the same returned audio.

For telephony, however, two different metrics decide whether a model is usable,
and both differ from that definition:

• speech-span RTF — synthesis time against speech duration. Silence
  padding inflates the denominator, so a run can be under 1.0 by the
  total-duration definition while still falling behind during actual speech.
• time to first audible speech — a TTFB that ends at the first PCM chunk
  is satisfied even when that chunk is silent. With a leading-silence p95 of
  3.675 s, first-chunk latency and first-audible-speech latency can differ by
  seconds.

These are our product metrics, not a claim that yours should change.

What would help

1. The fixture behind the published numbers: prompt text(s), reference clip, and
   the resulting audio-duration distribution.
2. The hardware and warmup state, and whether torch.compile was warm.
3. The number of runs and the aggregation (single best run, mean, or median).
4. Confirmation of whether the published RTF is computed against total returned
   audio duration or against speech duration.
5. Confirmation of what the 180 ms TTFB measures — first PCM chunk, or first
   chunk containing speech.
6. If possible, publishing the benchmark script so results are reproducible.

Happy to share our harness, raw WAVs, per-request JSONL and GPU telemetry if
that is useful for comparison.

Start by reviewing the model card, the benchmark commit, the upstream MOSS-TTS Transformers path, and the serving code that computes RTF and TTFB. Reproduce the stated fixture and compare total-duration versus speech-span RTF and first-chunk versus first-audible-speech latency. Done means the benchmark inputs, warmup state, aggregation, metric definitions, and reproducible script are documented.