slow inference with peft model

I use the code from fingpt benckmark to evaluate my peft model on datasets：fpb/fiqa/tfns respectively.
However the inference speed is very slow, and increasing batch size doesn't help with this issue.
It seems that when i increase the batch size, the inference speed per iteration increases consequently.
Could u please help with this issue?