Streaming ASR leaks previous-turn punctuation into the next final (leading ? / ,)

Summary

With streaming ASR endpointing enabled on Nemotron Speech Streaming 0.6B, trailing punctuation from one utterance is not always attached to that utterance's is_final result. It arrives later as its own final, or as a leading ? / , / . on the next utterance.

Example from a live gRPC session (riva_server in nvcr.io/nvidia/nemo-speech.cpp:0.1.0):

Transcription: [Hi, can you tell me what is the capital of UK]
(user turn ends)
Transcription: [ ? What is the capital]
Transcription: [ of United States of America]

The first utterance is missing the question mark. The next utterance starts with that leftover ?.

This is not a client transcript-sort issue. The pipeline concatenates is_final chunks in receive order. The punctuation is missing from the earlier final and present on the later one.

Repro

1. Serve ASR with endpointing on Nemotron Speech Streaming 0.6B (--asr.endpointing.enable=true, --asr.endpointing.stop_history_eou_ms=400).
2. Keep one streaming Recognize stream open across multiple spoken turns (Riva gRPC, as a voice-agent client does).
3. Ask a question that should end with ?, wait for the bot, then ask another question.
4. Repeat a few turns.

Also seen with a leading comma:

Transcription: [ , okay]
Transcription: [ ? And what is the capital]

Expected

Punctuation that belongs to utterance N is included in that utterance's final, or flushed before the next speech segment starts. The next final must not start with leftover ?, ,, or . from the previous turn.

Actual

The word text finalizes first without trailing punctuation. After the next user turn has already started, a later final carries that punctuation. Sometimes it is punctuation-only ([ ?]). Sometimes it is prepended to the next hypothesis ([ ? What is the capital]). A punctuation-only final can also start a new user turn on the client (TranscriptionUserTurnStartStrategy).

Observed timing (one session, UTC):

• 07:10:56.827 [ London] (no ?)

• 07:10:56.829 user stopped

• 07:11:00.482 next user started

• 07:11:05.343 [ ?]

• 07:12:19.572 [Hi, can you tell me what is the capital of UK] (no ?)

• 07:12:19.573 user stopped

• 07:12:23.288 next user started

• 07:12:24.853 [ ? What is the capital]

Root cause

Streaming endpointing is supposed to emit one final per utterance and reset decoder state so delayed tokens do not leak. Trailing punctuation is still emitted after that reset, on the same Recognize stream.

Likely the punctuation path (self-punctuation on Nemotron RNNT, or a delayed rewrite of . / , / ?) is not flushed with the EOU final. It is applied when more audio arrives, so it lands on the next segment.

The client cannot tell the leftover ? apart from real next-turn text, because both arrive as is_final on the same stream.

Environment

• nvcr.io/nvidia/nemo-speech.cpp:0.1.0
• Nemotron Speech Streaming EN 0.6B GGUF (nemotron-speech-streaming-en-0.6b.q8_0.gguf)
• Endpointing on, stop_history_eou_ms=400
• Persistent Riva gRPC streaming Recognize (Pipecat NvidiaSTTService)
• No separate --pnc-model passed. Punctuation is coming from the ASR path itself.

Request

Flush trailing punctuation with the utterance that produced it. Do not emit a later punctuation-only final, and do not prepend previous-turn ? / , / . onto the next hypothesis. Tracking this here so GGML riva_server and the streaming ASR client stay aligned.

Start with the riva_server streaming endpointing/finalization and punctuation paths, using the persistent Riva gRPC stream and the listed endpointing flags to reproduce the delayed punctuation. Done means trailing punctuation is included in the preceding utterance’s final, with no punctuation-only final or leading punctuation on the next hypothesis.