facebookresearch / facebookresearch/cowtracker

Clarification on evaluation protocol in Table 1

Open
#6 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
182
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Hi, thank you for your work. Exploring alternatives to correlation-based matching for point tracking is a valuable direction.

I have a question regarding the evaluation and comparison setup in the paper. I was unable to find a contact address for the first author, so I am posting here and hope this is an appropriate place.

In Section 4.2 (Table 1), dense and sparse trackers are compared for point tracking. The reported numbers for CoTracker3 and AllTracker appear consistent with those in their respective papers. However, there seems to be a potential inconsistency in the evaluation protocol:
* CoTracker3 reports results using queries from all frames (t >= 0).
* AllTracker reports results using queries from the first frame only (t = 0).

This difference is also reflected in the AllTracker paper, where CoTracker3 is re-evaluated under the t = 0 setting, leading to differences with its original reported numbers.

Based on this, it appears that the current comparison table may include t = 0 evaluations for AllTracker, while CoTracker3 (and other sparse trackers) are evaluated under t >= 0.

Could you clarify whether CoWTracker is evaluated using queries from the first frame only (t = 0), or from all frames (t >= 0)? This distinction would impact the interpretation of the reported results.

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with Section 4.2 and Table 1 in the paper, then compare the cited CoTracker3 and AllTracker evaluation protocols. Determine whether CoWTracker uses queries from t = 0 or t >= 0, and document the clarification or resulting comparison adjustment so the reported results are interpreted consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.