facebookresearch / facebookresearch/co-tracker

cannot backward tracking in online model with specific queries in middle frame

Open
#60 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
5.1k
Forks
389
PR merge metrics
No merged PRs in 30d

Description

Hi,

I am using your online model for point tracking. and I want to process a video. I want to set up specific queries in one of the middle frame and try to use backward tracking. here is my code:

`video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device)
cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2_online").to(device)
cotracker(video_chunk = video, is_first_step = True, queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True)
vis = Visualizer(save_dir="./co-tracker/result", pad_value=120, linewidth=2)
for ind in range(0, video.shape[1] - cotracker.step, cotracker.step):
pred_tracks, pred_visibility = cotracker(video_chunk=video[:, ind : ind + cotracker.step * 2], queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True)

vis.visualize(video, pred_tracks, pred_visibility)`

and here is the bug i got:
---> 60 pred_tracks, pred_visibility = cotracker(video_chunk=video[:, ind : ind + cotracker.step * 2], queries=maxima_coords[None], grid_query_frame=grid_query_frame, backward_tracking=True)
61
62 vis.visualize(video, pred_tracks, pred_visibility)

~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1129 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130 return forward_call(*input, **kwargs)
1131 # Do not call functions when jit is used
1132 full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/lib/python3.9/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
25 def decorate_context(*args, **kwargs):
26 with self.clone():
---> 27 return func(*args, **kwargs)
28 return cast(F, decorate_context)
29

TypeError: forward() got an unexpected keyword argument 'backward_tracking'

Contributor guide

Open the contributing guide

Research direction

Start by checking the forward signature and implementation of the torch.hub-loaded cotracker2_online model, focusing on the provided backward_tracking and grid_query_frame arguments. Reproduce the TypeError with the supplied video-chunk loop, then confirm that the intended middle-frame backward-tracking workflow is supported and covered by a runnable example or regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.