mouse interaction causes emacs to stop responding to keys

**Describe the bug**
using the mouse on any part of the eaf buffer or using it outside of emacs while having the eaf window focused makes
key events to emacs not responding or produce different behaviour until i use the mouse to select other buffer than
eaf buffer.

**To Reproduce**
1. M-x browse-url www.google.com
2. use the mouse on any part of the eaf buffer or switch to other window and use the mouse
3. try to use the keyboard keys and most of the keys dont work
4. sometimes the keys work but with different behaviour (for ex: j zooms in and k zooms out)

**Expected behavior**
keys keep working while being able to use the mouse

**Versions (please complete the following info):**
- Distro and DE/WM: wmctrl -m
Name: GNOME Shell
Class: N/A
PID: N/A
Window manager's "showing the desktop" mode: OFF

- Versions of Dependencies:
- pyqt5: 5.14.1+dfsg-3build1
- wmctrl: 1.07-7build1
- node: 16.9.0-deb-1nodesource1
- npm: 7.21.1
- libglib2: 2.64.6-1~ubuntu20.04.4
- python-sip: 4.19.21+dfsg-1build1
- python3-pyqt5.qtwebengine: 5.14.0-2build1
- M-x emacs-version: GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0) of 2022-01-01 (also tested on 29 and i had the same behaviour )

**Additional context**
I use evil and evil states `in the eaf buffer to be able to use my normal mode keybindings to switch windows and so on.