[Issue]: Whisper periodic forced timestamp sampling truncates Japanese sentence endings

Problem Description

I found a reproducible Japanese sentence-ending truncation issue in Whisper V3 Turbo running through FastFlowLM v0.9.45 on a Ryzen AI NPU.

The strongest cause isolated in controlled A/B testing was the decoder watchdog that periodically forces sampling from the timestamp-token range after approximately 16 decoded tokens:

if (watching_dog == 0 && allow_force_time_stamp) {
    last_idx = this->_sample_in_time_stamp(logits);
    watching_dog = 16;
}

This code is also present in current main at commit 2d6c4838a02555478f0de1e57fe6428e614b67ba.

Disabling only this periodic forced timestamp selection recovered all three reproducible sentence endings in the original Japanese test set. It does not disable normal Whisper timestamp tokens.

Aggregate results:

Feeding the detected language token into the decoder KV cache was tested independently and did not improve CER in this test set. Disabling the watchdog exposed a separate exact-digital-silence repetition case, so the candidate returns an empty result only when every PCM sample is exactly 0.0f; it does not use a near-silence threshold.

Full methodology, limitations, aggregate measurements, and the v0.9.45 patch:

https://github.com/tamayanaohito/fastflowlm-whisper-japanese-decoder-fix

This is an experimental report, not a claim of general Japanese ASR improvement. Human speech coverage is limited to one speaker and one recording. Personal audio and raw transcripts are not published.

Would the maintainers consider removing, revising, or making this periodic forced timestamp watchdog configurable? I can rebase the minimal change onto current main if that direction is acceptable.

Operating System

Microsoft Windows 11 Home, version 10.0.26200, build 26200

CPU

AMD Ryzen AI 9 465 with Radeon 880M

GPU

AMD Radeon 880M and NVIDIA GeForce RTX 5060 Laptop GPU are installed, but neither was used for ASR inference. Whisper inference ran on the Ryzen AI NPU.

ROCm Version

N/A on Windows. FastFlowLM used the Ryzen AI NPU backend with XRT 2.21.75.

ROCm Component

No response

Steps to Reproduce

1. Check out FastFlowLM v0.9.45 (f43272c77bb6de4984058430b3e5fbf3e7926c61) and build an unchanged baseline.
2. Transcribe short Japanese WAV files whose reference text extends beyond approximately 16 decoded tokens.
3. Confirm that the rebuilt baseline matches the installed FastFlowLM result on identical WAV bytes.
4. Apply fastflowlm-v0.9.45-whisper-decoder.patch from the linked repository.
5. Run the candidate in a separate process with HIGHRISE_WHISPER_DECODE=natural-timestamps.
6. Compare complete output and normalized CER using identical WAV bytes and serialized requests.
7. Test exact digital silence and quiet non-zero speech separately.

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

Paste output here

Additional Information

• Model: Whisper V3 Turbo / whisper-v3:turbo
• Inference device: Ryzen AI NPU
• FastFlowLM target: v0.9.45; current main source was also inspected
• Requests were serialized and each mode ran in a separate loopback-only process
• Model files were hash-checked before and after testing
• A separately loaded NPU vision model remained ready during the comparisons
• No NPU kernels, model weights, audio preprocessing, public APIs, or postprocessing were changed

Start at the decoder watchdog code containing the periodic sample_in_time_stamp call and compare current main with the v0.9.45 behavior. Reproduce the Japanese WAV cases using the listed serialized-request steps, then compare complete output and CER with and without the watchdog. Done means an agreed watchdog behavior avoids the truncation without regressing quiet speech or exact digital silence.