Evaluation differences compared to prior work
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 3k
- Forks
- 234
- PR merge metrics
- No merged PRs in 30d
Description
Thanks for your great work, the results are amazing!
Just curious why the evaluation tables in MoGe often have different baseline numbers than the numbers reported in the original papers?
Here are some examples:
DUSt3R comparisons on NYUv2
In MoGe paper, Table 2 (scale-invariant pointmap): Rel 5.56, Delta1 97.1
In MoGe paper, Table 2 (affine-invariant pointmap): Rel 4.49, Delta1 97.4
In MoGe paper, Table 3 (scale-invariant depth): Rel 4.43, Delta1 97.1
In DUSt3R paper, Table 2 (depth): Rel 6.50, Delta 94.09
DUSt3R comparisons on KITTI
In MoGe paper, Table 2 (scale-invariant pointmap): Rel 21.9, Delta1 63.6
In MoGe paper, Table 2 (affine-invariant pointmap): Rel 18.0, Delta1 66.7
In MoGe paper, Table 3 (scale-invariant depth): Rel 7.71, Delta1 90.9
In DUSt3R paper, Table 2 (depth 512): Rel 10.74, Delta 86.60
Marigold comparisons on NYUv2
In MoGe paper, Table 3 (affine-invariant depth): Rel 4.63, Delta1 97.3
In Marigold paper, Table 1 (depth w/ ensemble): Rel 5.5, Delta1 96.4
Marigold comparisons on KITTI
In MoGe paper, Table 3 (affine-invariant depth): Rel 7.29, Delta1 93.8
In Marigold paper, Table 1 (depth w/ ensemble): Rel 9.9, Delta1 91.6
Marigold comparisons on ETH3D
In MoGe paper, Table 3 (affine-invariant depth): Rel 6.08, Delta1 96.3
In Marigold paper, Table 1 (depth w/ ensemble): Rel 6.5, Delta1 96.0
Marigold comparisons on DIODE
In MoGe paper, Table 3 (affine-invariant depth): Rel 6.34, Delta1 94.3
In Marigold paper, Table 1 (depth w/ ensemble): Rel 30.8, Delta1 77.3
DepthAnythingV2 comparisons on Sintel
In MoGe paper, Table 3 (affine-invariant disparity): Rel 21.4, Delta1 72.8
In DepthAnythingV2 paper, Table 5 (take their best result): Rel 48.7, Delta1 75.2
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the MoGe paper's Tables 2 and 3 with the cited DUSt3R, Marigold, and DepthAnythingV2 tables, starting with their dataset, metric, prediction type, and evaluation-protocol descriptions. Determine which protocol differences explain the reported values, then document the explanation and any needed corrections in the project’s issue or evaluation documentation.
Written by the indexing model from the issue text.
Assessment
- Domain
- computer-vision, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100