Guest not responding to keyboard properly after switching to another app

M1 Mac, UTM 2.1.2 (App Store version), MacOS 11.5.1

I have only used Debian based guests with UTM so I don't know if this is specific to Debian or Linux or all kinds of VM's. I regularly can't seem to type anymore when I switch back to from the host to the VM, I have to restart the VM and then everything works fine again. Sometimes I also have the issue that hitting the p key will launch display settings in the VM instead of typing the p.