speech recognization problem while making a call on speakerphone

I'm using vosk freeswitch plugin to do speech recognization, and playAndGetDigits function to play an audio to user while doing speech recognization. There is a problem that while user is using speakerphone to make the call and is keeping quiet, the audio played will be recoginized. How can I omit the audio played and only detect the user speaking? Thanks!