home-assistant / home-assistant/core

assist_pipeline: microVAD's -1.0 warm-up value is compared as a speech probability, so short commands never endpoint and run to the 15s timeout

Open
#181,747 1 comment 0 reactions 2 assignees View on GitHub

@arturpragacz is already working on this.

Since Sep 9, 2026.

integration: assist_pipeline
Dominant language
Python
Stars
90.8k
Forks
38.7k
Avg merge
1d 2h
Merged PRs (30d)
597

Description

### The problem

`MicroVad.Process10ms` returns **`-1.0` for the first 760ms of any stream** — I
measured it as identical for digital silence, room tone, continuous speech and a
1kHz tone, so it is a "no answer yet" sentinel rather than a measurement.

`VoiceCommandSegmenter.process` compares it directly:

```python
if speech_probability is None:
speech_probability = 0.0
...
is_speech = speech_probability > self.before_command_speech_threshold
```

`None` is normalised; `-1.0` is not. So the warm-up counts as **silence**, while
still decrementing `_timeout_seconds_left`.

The consequence is that the first 760ms of every stream can contribute nothing
toward the 0.3s of speech needed to set `in_command`. A command that is over, or
nearly over, by then never gets there — `STT_VAD_START` is never emitted, and
the only remaining exit is `timeout_seconds = 15.0`. Short commands ("stop",
"yes") therefore take ~15 seconds while longer ones answer normally, and no
amount of microphone gain or filtering changes it.

This is the symptom in #122177 (@HarvsG, July 2024), which was closed by the
stale bot in February 2025 without a diagnosis. That report predates the
microVAD switch in #122861, so the arithmetic above is not what he was seeing,
but the file and the symptom are the same and it is still happening.

**Clients cannot tell this from a normal endpoint.** `process()` sets
`self.timed_out`, added with tests in #123942, but nothing reads it — searching
the repository, it appears only in `vad.py` itself. `_speech_to_text_stream`
emits the same `STT_VAD_END` either way, so a satellite sees a successful
end-of-speech.

To be clear, I am not asking to undo #132987 — a timeout is a normal enough
event that a user-facing warning would be noise, and I agree with that call. The
gap is that there is no machine-readable signal either, so a satellite cannot
distinguish "the user stopped talking" from "our VAD gave up 15 seconds ago" and
act accordingly.

`timeout_seconds` is also fixed: `_speech_to_text` passes only `silence_seconds`
when constructing the segmenter. This is why changing "finished speaking
detection" doesn't help — that moves `silence_seconds`, and on these turns the
segmenter never enters the command state for silence to end.

## Reproduction

No hardware. This drives the shipped segmenter and the pinned
`pymicro-vad==1.0.1`, chunked as `_speech_to_text_stream` does:

```python
import wave
from pymicro_vad import MicroVad
from homeassistant.components.assist_pipeline.vad import VoiceCommandSegmenter

def run(path): # 16kHz mono s16le: a short command,
pcm = wave.open(path).readframes(10**9) # then several seconds of quiet
vad, seg = MicroVad(), VoiceCommandSegmenter()
for i in range(0, len(pcm) - 320 + 1, 320): # 10ms chunks
ms = i // 320 * 10
if not seg.process(0.01, vad.Process10ms(pcm[i:i + 320])):
return f"ended {ms}ms, timed_out={seg.timed_out}"
if seg.in_command:
return f"STT_VAD_START {ms}ms"
return "stream ended, in_command never set"
```

A 0.45s "stop" and a 0.86s "what's the weather", each in 16s of room tone:

```
short "stop" : ended 15000ms, timed_out=True
"what's the weather" : STT_VAD_START 1050ms
```

The short one never sets `in_command` at all. Leading silence does
not help: I swept 0/240/480/760/1000ms of lead over 40 noise seeds each and the
short command timed out 40/40 at every lead.

## Field data

845 wake-word turns across my own devices over a month, timed from the
pipeline's own `STT_VAD_END`:

| end of speech | turns |
|---|---|
| 14.50s | 2 |
| 14.75s | 0 |
| 15.00s | 2 |
| **15.25s** | **24** |
| 15.50s | 1 |

27 of 845 (3.2%) in that spike, every one carrying exactly 15,120ms of audio,
about half returning no recognised text. A spike in one 250ms bin against 0–3 in
its neighbours is a fixed cap firing rather than long commands.

On my own satellite, which trims 240ms from the front of the stream to remove
the wake-word tail, the word "stop" hit the cap on the wake-word path and
endpointed normally at 3.7s on the follow-up path, which does not trim. Same
word, same room, same device.

### What version of Home Assistant Core has the issue?

2026.9.1

### What was the last working version of Home Assistant Core?

Not known to have worked. The mechanism above dates from the microVAD switch in 2024.8; #122177 reports the same symptom on 2024.7.2.

### What type of installation are you running?

Home Assistant OS

### Integration causing the issue

assist_pipeline

### Link to integration documentation on our website

https://www.home-assistant.io/integrations/assist_pipeline

### Diagnostics information

_No response_

### Example YAML snippet

_No response_

### Anything in the logs that might be useful for us?

At debug level, `homeassistant.components.assist_pipeline.vad` logs
`VAD end of speech detection timed out after 15.0 seconds` on each
affected turn. Nothing at info or above, by design since #132987.

### Additional information

## Suggested direction

I'd be glad to open a PR for either of the first two; the third is a tuning
question I don't want to guess at, given #134360.

- **Treat a negative `speech_probability` as "no answer"**, as `None` already
is, rather than comparing it against the speech threshold.
- **Expose `timed_out` to clients** — in the `STT_VAD_END` event data, or as a
distinct event — so satellites and the debug trace can tell the two apart.
Not as a log line; see above.
- Possibly don't start `_timeout_seconds_left` until the VAD is answering, so
the 15s budget measures the user rather than the model's warm-up.

Thanks to @HarvsG for reporting the symptom and locating it in this file long
before I ran into it.

---

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.