Voice Assistant stops working-listening

### The problem

Hi guys. Today I have faced this issue on my [Atom Echo](https://shop.m5stack.com/products/atom-echo-smart-speaker-dev-kit),

```
[20:08:20][D][esp-idf:000]: I (23729714) I2S: DMA Malloc info, datalen=blocksize=1024, dma_buf_count=4

[20:08:20][D][voice_assistant:422]: State changed from STARTING_MICROPHONE to WAIT_FOR_VAD
[20:08:20][D][voice_assistant:176]: Waiting for speech...
[20:08:20][D][voice_assistant:422]: State changed from WAIT_FOR_VAD to WAITING_FOR_VAD
[20:08:20][D][voice_assistant:189]: VAD detected speech
[20:08:20][D][voice_assistant:422]: State changed from WAITING_FOR_VAD to START_PIPELINE
[20:08:20][D][voice_assistant:428]: Desired state set to STREAMING_MICROPHONE
[20:08:20][D][voice_assistant:206]: Requesting start...
[20:08:20][D][voice_assistant:422]: State changed from START_PIPELINE to STARTING_PIPELINE
[20:08:20][D][voice_assistant:443]: Client started, streaming microphone
[20:08:20][D][voice_assistant:422]: State changed from STARTING_PIPELINE to STREAMING_MICROPHONE
[20:08:20][D][voice_assistant:428]: Desired state set to STREAMING_MICROPHONE
[20:08:20][D][voice_assistant:529]: Event Type: 1
[20:08:20][D][voice_assistant:532]: Assist Pipeline running
[20:08:20][D][voice_assistant:529]: Event Type: 9
[20:08:20][D][voice_assistant:529]: Event Type: 99
[20:08:20][D][voice_assistant:675]: TTS stream end
```

As you can see on `tts_stream_end` my m5 stack echo stops listening and I have to turn off and on the wake word to make it work, at this time I can see the log working again changing the listening states, but when I say an intent this thing happens again. Do you know what is happenning?

The text to speech doesn’t work properly either. The m5 stack echo stops the speech in the middle of the response. I’m using NabuCasa service.

### Which version of ESPHome has the issue?

2023.11.6

### What type of installation are you using?

Home Assistant Add-on

### Which version of Home Assistant has the issue?

2023.12.0

### Board

ESP32 M5 stack atom echo

Start by tracing the voice_assistant state transitions around WAIT_FOR_VAD, STREAMING_MICROPHONE, and the logged TTS stream end on an Atom Echo using ESPHome 2023.11.6. Reproduce the failure with the listed Home Assistant and Nabu Casa setup; done means listening resumes after TTS and the full speech response plays without requiring the wake word to be toggled.

Written by the indexing model from the issue text.