Removing Queries as the tracking progresses

Hey, I'm amazed at how well this works, however I have an issue with implementing this for my use case.

In my scenario, both the camera and the points to track are moving, this makes it fairly hard, I do realize that, but its still works well.

My issue is, that as the video progresses, points that have long been out of frame get "re-detected" in absurd or weird locations because they roughly match the color of the last tracked point.

In my opinion, it would be helpful to have a way to remove or mark queries as "do not track further", or any other way to stop tracking them. I tried adjusting the queries mid-processing, but this did not work, made obvious by this code:
[predictor.py](https://github.com/facebookresearch/co-tracker/blob/main/cotracker/predictor.py#L242)
showing it's only set on the first step.

Any ideas? Could I just modify the code to set queries even if it's not the first step, or will this break things?

Start in cotracker/predictor.py around line 242 and trace how queries are initialized and used after the first processing step. Reproduce the moving-camera case described in the issue, then define completion as allowing an individual query to stop being tracked without later re-detection at a new location.

Written by the indexing model from the issue text.