esphome / esphome/feature-requests

Microphone Resampler Component (to cope with a shared I2S bus)

Open
#3,072 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
450
Forks
29
PR merge metrics
No merged PRs in 30d

Description

**Describe the problem you have/What new integration you would like**

The [Onju Voice](https://github.com/justLV/onju-voice) board I'm working with supports only one I2S bus for the microphones and speaker. As I'd like to use the on-device MircoWakeWord engine, which requires a 16khz input signal (see [this code line](https://github.com/esphome/esphome/blob/1d6d0d66dca39d1fedf7b2496da4e19835679a12/esphome/components/micro_wake_word/preprocessor_settings.h#L15) and [this code line](https://github.com/esphome/home-assistant-voice-pe/blob/4b31449b8b6df943320ba91d476b7edc488f7d82/esphome/components/micro_wake_word/preprocessor_settings.h#L19)), the whole I2S bus must run at 16khz (see [this comment](https://github.com/tetele/onju-voice-satellite/issues/52#issuecomment-2225751543) and [this comment](https://github.com/tetele/onju-voice-satellite/discussions/113#discussioncomment-12349993) and [this comment](https://github.com/esphome/issues/issues/6599#issuecomment-2667595576) and [this discussion](https://github.com/tetele/onju-voice-satellite/discussions/100#discussion-7673704)). This is no issue to audio input, but decreases the perceived quality of audio output (e.g. music through a media_player), which is now restricted to 16khz as well.

The recently introduced resample speaker (see [this PR](https://github.com/esphome/esphome/pull/8169)) offers to resample audio output up and down. Thus, I think a similar resample microphone is desirable, which could reuse most of the code of the resample speaker and underlying resampler implementation (see [this repo](https://github.com/esphome/esp-audio-libs/tree/main)). This would allow to resample the received audio from the I2S bus (48kHz) to 16kHz before publishing it as output of the microphone. Thus, the chain would be i2s_input -> resampler -> microphone, while the resampler carries over any needed settings from the microphone (like the d_in_pin).

While downsampling seems to be computationally expensive (see "facts" in [this discussion](https://github.com/tetele/onju-voice-satellite/discussions/100#discussion-7673704)), I have not seen an implemented experiment yet, that allows to draw the conclusion that is too expensive for an ESP32S3. In contrast, I've used an experiment that implemented downsampling, which works just fine on an ESP32S3 (see the non-standard approach below).

**Please describe your use case for this integration and alternatives you've tried:**

Use Case: Decent audio output quality (at least 44.1khz) from a shared I2S bus (speaker and microphone sharing the bus), while allowing to utilize the on-device MicroWakeWord engine (requiring 16khz).

A non-standard/-stable approach (see [this code](https://github.com/dreimer1986/onju-voice-satellite/blob/cc87ace008a7d7f0b6b7ceba3aac8d984b31819e/esphome/onju-voice-microwakeword.yaml#L171) and [this repo](https://github.com/gnumpi/esphome_audio/tree/dev-next)) allows setting the I2S bus to 48khz while resampling the audio input from the bus to 16khz before making it available as output of the esphome microphone, effectively meeting the requirements of the MicroWakeWord engine. I got this setup running and it works as expected. The I2S bus is now running at 48khz, which restores the perceived output audio quality to "good". Nevertheless, this specific code approach was abandoned, it requires the v4 esp-idf, and isn't usable with the recent speaker and media_player PRs from 2025.02 (requiring to fall back to at least 2024.12).

**Additional context**

According to some discussions I read ([here](https://github.com/esphome/esphome/pull/8181#issuecomment-2703693786), [here](https://github.com/tetele/onju-voice-satellite/discussions/112#discussioncomment-12325454), and [here](https://github.com/orgs/FutureProofHomes/discussions/6)), it should be possible to tune esphome's I2S implementation for dynamically configuring it to either 16kHz or 48kHz based on current requirements. Unfortunately, this appears to be rather complex and one would need to touch many components due to breaking changes. Thus, an input resampler seems to be the easier option or at least an intermediate option until someone reworks the I2S implementation.

EDIT: Of course, this still requires reinitializing the i2s bus for every action (recording through mic or playing on speaker), as esphome's current i2s implementation won't support duplex mode. See [here](https://github.com/tetele/onju-voice-satellite/pull/108#issuecomment-2704111636).

Code example of an envisioned config:

```yaml
i2s_audio:
- id: i2s_shared
i2s_lrclk_pin:
number: GPIO...
i2s_bclk_pin:
number: GPIO...

speaker:
- platform: i2s_audio
id: i2s_audio_speaker
sample_rate: 48000
i2s_dout_pin: GPIO...
bits_per_sample: 32bit
i2s_audio_id: i2s_shared
...

- platform: resampler
id: resampling_microphone
input_bus: i2s_shared
i2s_sampling_rate: 48000
i2s_bits_per_sample: 32bit
# output sampling_rate inherited from downstream microphone
# output bits_per_second inherited from downstream microphone

microphone:
- platform: nabu_microphone # from voice-pe
i2s_din_pin: GPIO...
...
sample_rate: 16000 # output_sample_rate
bits_per_sample: 32bit # output_bits_per_second
i2s_audio_id: resampling_microphone
channel_0:
id: asr_micr
channel_1:
id: mww_mic

micro_wake_word: # from voice-pe
id: mww
models: ...
vad: ...
microphone: mww_mic

voice_assistant: # from voice-pe
id: va
microphone: asr_mic
...
```

The following people might be interested in this component: @tetele , @dreimer1986 , @kahrendt , @jhbruhn

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.