Batching _run_single_frame_inference

In the video predictor there is a comment stating:
```
# We skip those frames already in consolidated outputs (these are frames
# that received input clicks or mask). Note that we cannot directly run
# batched forward on them via `_run_single_frame_inference` because the
# number of clicks on each object might be different.
```

In my usecase I can guarantee only a single input click per object. Would I be able to batch the pipeline even further? If so, could you give some pointers on where best to start?
Thanks!