Issue: CoTracker Queries Not Tracking Correctly

Hi Team,

I'm encountering an issue with CoTracker. When I use the queries generated by seg_to_queries, the tracking does not work as expected. Below is the relevant code snippet and output details:

```
DEFAULT_DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
seg_file = '/home/richal/cotracker_code/video41/video_41_mask/00001.png'
queries = seg_to_queries(seg_file, frame_index=0, n=10, max_objects=10, background_id=0, device=DEFAULT_DEVICE)
visualize_queries(seg_file, queries, output_file="queries_visualization_video_mask.png")
is_first_step = True
grid_size = 30,
grid_query_frame = 1,

# Iterating over video_input frames, processing one window at a time:

for i, frame in enumerate(
iio.imiter(
video_path,
plugin="FFMPEG",
)
):
if i % cotracker.step == 0 and i != 0:
pred_tracks, pred_visibility = _process_step(
window_frames,
is_first_step,
queries = queries,
grid_size=grid_size,
grid_query_frame=grid_query_frame,
)
is_first_step = False
window_frames.append(frame)

# Processing the final video_input frames in case video_input length is not a multiple of model.step
pred_tracks, pred_visibility = _process_step(
window_frames[-(i % cotracker.step) - cotracker.step - 1:],
is_first_step,
queries=queries,
grid_size=grid_size,
grid_query_frame=grid_query_frame,
)
```

Output details:
Queries tensor shape:
`torch.Size([1, 21, 3])`
Queries tensor values:

```
[[[ 0 379 106]
[ 0 379 266]
[ 0 95 106]
[ 0 95 160]
[ 0 190 160]
[ 0 284 160]
[ 0 379 160]
[ 0 95 213]
[ 0 190 213]
[ 0 284 213]
[ 0 379 213]
[ 0 95 266]
[ 0 95 319]
[ 0 284 319]
[ 0 284 373]
[ 0 474 53]
[ 0 758 213]
[ 0 474 266]
[ 0 569 266]
[ 0 663 266]
[ 0 758 266]]]
```
Predicted tracks and visibilities are computed but do not reflect correct tracking.
Predicted tracks:
```
[[[[3.8339970e-01 9.6302666e+01]
[2.1980672e-01 2.4304851e+02]
[3.8339970e-01 9.6302666e+01]
...
[2.1980672e-01 2.4304851e+02]
[2.1980672e-01 2.4304851e+02]
[2.1980672e-01 2.4304851e+02]]

[[5.4503083e-01 9.6144287e+01]
[4.1200501e-01 2.4287111e+02]
[5.4503083e-01 9.6144287e+01]
...
[4.1200501e-01 2.4287111e+02]
[4.1200501e-01 2.4287111e+02]
[4.1200501e-01 2.4287111e+02]]

[[5.7194012e-01 9.5992401e+01]
[5.0720090e-01 2.4278004e+02]
[5.7194012e-01 9.5992401e+01]
...
[5.0720090e-01 2.4278004e+02]
[5.0720090e-01 2.4278004e+02]
[5.0720090e-01 2.4278004e+02]]

```
Could you please look into this issue? Any guidance or fixes would be greatly appreciated.

Thanks,
Chenhao

Start by reproducing the reported flow using seg_to_queries on /home/richal/cotracker_code/video41/video_41_mask/00001.png, then trace how queries are passed through _process_step while iterating video_input. Compare the resulting pred_tracks and pred_visibility with the video and query visualization to identify where tracking diverges; done means the generated queries produce correct tracks.

Written by the indexing model from the issue text.