Reduce latency when using Rnnlm rescoring

Dear all,
I have some questions about RNNLM rescoring. I have compared the latency of 3 type of decoding (1-pass, carpa rescoring and rnnlm rescoring) and found that the latency of Rnnlm rescoring is really high than the others. I've tried to modify the source code of Vosk in recognizer.cc file to improve the rnnlm rescoring but getting no better result.

Therefore, is there any way to reduce the latency when using Rnnlm rescoring? If yes, can you list some directions for me to follow. I think we can modify the source code of Vosk to reduce the latency when using Rnnlm. The accuracy is not very important for me, just how to reduce the latency when using rnnlm rescoring.

Best regards