Problem with bridge mode with remote desktop

To be able to test our devices which doesn't have USB host, we are using pywalt bridge mode, and forward the mort using adb reverse.
This needed to change the IP in APK source code to 127.0.0.1

Doing this, we hit some clock issues.
Checking the drift, or forcing re-sync of clocks doesn't seem to raise anything. Latency is always around 120us.

This has been tested on an Android 5.1 phone, both in USB Host mode and adb reverse bridged mode.
USB host mode works properly, adb reverse bridged mode raises negative dt, all events are dropped because of this.

Sometimes we notice small abs(dt) like -20, but sometimes it looks like the dt is coming from previous tap.
For instance if we wait ~ 15s between two taps, we notice a ~15s dt.

Have you got some ideas on where to look at?