Remote Control API

We're currently using scrcpy to perform a series of QA tests. We use it to stream the screen content in one direction and user inputs in the other direction. While this works well, the tasks are quite repetive and so we started writing some automations for this.

Something like: If (screen shows x) then click(x, y).

To do so we have written an external software that copies the screen content from scrcpy and eventually send events like mouse clicks to it. While this works more or less sufficient for our needs, it still lacks some features. For example, we must keep the scrcpy window shown (i.e. not minimized) and the user must keep the mouse still while the automation is in progress. Also, we haven't found a way to input multi touch gestures.

It would be very handy if the scrcpy comes as library (or with a library) or exports that would allow a programmatic extraction of data and supplying input data to it.

Any thoughts about it? It shouldn't be that complicated and I'm willing to help, but at the moment I have no idea how much work this will be in the end or whether this is already possible in some way.

Start by reviewing scrcpy's existing screen-streaming and input paths, then compare them with the automation needs described here. Done would mean exposing a programmatic way to extract screen data and supply input, including multitouch, without requiring a visible window or a stationary mouse.

Written by the indexing model from the issue text.