microsoft / microsoft/MoGe

Evaluation differences compared to prior work

Open
#11 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
3k
Forks
234
PR merge metrics
No merged PRs in 30d

Description

Thanks for your great work, the results are amazing!

Just curious why the evaluation tables in MoGe often have different baseline numbers than the numbers reported in the original papers?

Here are some examples:

DUSt3R comparisons on NYUv2
In MoGe paper, Table 2 (scale-invariant pointmap): Rel 5.56, Delta1 97.1
In MoGe paper, Table 2 (affine-invariant pointmap): Rel 4.49, Delta1 97.4
In MoGe paper, Table 3 (scale-invariant depth): Rel 4.43, Delta1 97.1
In DUSt3R paper, Table 2 (depth): Rel 6.50, Delta 94.09

DUSt3R comparisons on KITTI
In MoGe paper, Table 2 (scale-invariant pointmap): Rel 21.9, Delta1 63.6
In MoGe paper, Table 2 (affine-invariant pointmap): Rel 18.0, Delta1 66.7
In MoGe paper, Table 3 (scale-invariant depth): Rel 7.71, Delta1 90.9
In DUSt3R paper, Table 2 (depth 512): Rel 10.74, Delta 86.60

Marigold comparisons on NYUv2
In MoGe paper, Table 3 (affine-invariant depth): Rel 4.63, Delta1 97.3
In Marigold paper, Table 1 (depth w/ ensemble): Rel 5.5, Delta1 96.4

Marigold comparisons on KITTI
In MoGe paper, Table 3 (affine-invariant depth): Rel 7.29, Delta1 93.8
In Marigold paper, Table 1 (depth w/ ensemble): Rel 9.9, Delta1 91.6

Marigold comparisons on ETH3D
In MoGe paper, Table 3 (affine-invariant depth): Rel 6.08, Delta1 96.3
In Marigold paper, Table 1 (depth w/ ensemble): Rel 6.5, Delta1 96.0

Marigold comparisons on DIODE
In MoGe paper, Table 3 (affine-invariant depth): Rel 6.34, Delta1 94.3
In Marigold paper, Table 1 (depth w/ ensemble): Rel 30.8, Delta1 77.3

DepthAnythingV2 comparisons on Sintel
In MoGe paper, Table 3 (affine-invariant disparity): Rel 21.4, Delta1 72.8
In DepthAnythingV2 paper, Table 5 (take their best result): Rel 48.7, Delta1 75.2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the MoGe paper's Tables 2 and 3 with the cited DUSt3R, Marigold, and DepthAnythingV2 tables, starting with their dataset, metric, prediction type, and evaluation-protocol descriptions. Determine which protocol differences explain the reported values, then document the explanation and any needed corrections in the project’s issue or evaluation documentation.

Written by the indexing model from the issue text.

Assessment

Domain
computer-vision, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.