Google cloud speech to text: huge latency (2~5sec)

We have been using google ASR.

Code : https://github.com/fonoster/fonoster/blob/main/mods/googleasr/src/tracker.ts

Instead of real time result, latency lies in range of 2s to 5s detoriating user experience. Any recommendations would be helpful!