google-deepmind / google-deepmind/physics-IQ-benchmark
Artifacts in 0112, 0113, 0114 verified videos
- Dominant language
- Python
- Stars
- 343
- Forks
- 47
- Avg merge
- 5h 11m
- Merged PRs (30d)
- 4
Description
It seems that for the three viewpoints (0112, 0113, 0114) in the "match blows balloon" scenario the artifact removal step for the "verified" set may have introduced other artifacts (see screenshot of the 'testing' split of 0014). Specifically, a light-colored box around scene probably intended to edit the video to stop the rotating platform in the 'deterministic artifacts' case. I do not see these artifacts for 0057 or 0051 with a similar rotating platform.
- Can you confirm that the artifacts I am seeing in these videos exist in the dataset and it is not a download error on my end? I re-downloaded the set multiple times and still see the artifacts. I do not see the artifacts in the original physics IQ videos.
- If these artifacts do exist in the dataset, would you be able to release a corrected version for these videos? I understand the importance of keeping the physics IQ verified evaluations stable, but I have other use cases for the dataset where having naturalistic videos (no artifacts like this) is important.
Thank you for this dataset and for your help!
Contributor guide
Research direction
Inspect the verified videos for viewpoints 0112, 0113, and 0114 in the “match blows balloon” scenario, especially the testing split, and compare them with the original physics IQ videos and similar rotating-platform videos 0057 and 0051. Confirm whether the light-colored box is present in the dataset rather than caused by download corruption; done means documenting the finding and determining whether corrected videos can be released.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100