facebookresearch / facebookresearch/Ego4d

Body pose looks misaligned in EgoExo4D

Open
#331 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
646
Forks
64
PR merge metrics
No merged PRs in 30d

Description

It seems that the human poses are not correct.
I'm simply plotting the 2D annotations as released.
This is take uid eba56e4f-7ec8-4d47-9380-e69928323e94 (iiith_cooking_111_2)

![image](https://github.com/facebookresearch/Ego4d/assets/6182426/899ac55f-4600-4b94-a14c-1dda1f367ea1)

I've tried a 2-3 other takes, too, and none seem correct.

Here is unc_basketball_03-31-23_02_14 cef2f19f-ec48-410c-8205-4572cc8706d9 frame 39
![image](https://github.com/facebookresearch/Ego4d/assets/6182426/6ee2fca2-d02b-4e0b-a268-20e7dc4590d0)

This visualization is directly of the 2D coordinates, I'm not projecting the 3D coordinates to the image by myself here (although that also results in misaligned projections).

Is this known?

Furthermore, when I estimate 2D poses on my own, per camera, and try to triangulate it myself using the given camera calibration parameters, it doesn't really work (the reprojection error is high), which leads me to suspect that the camera calibration may have issues.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the plotted 2D annotations for take eba56e4f-7ec8-4d47-9380-e69928323e94 and frame 39 of take cef2f19f-ec48-410c-8205-4572cc8706d9. Compare those results with projections from the supplied 3D coordinates and camera calibration, then verify whether triangulation reprojection error confirms an annotation or calibration problem. Done means identifying and documenting the confirmed source of the misalignment.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
computer-vision, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.