google-deepmind / google-deepmind/physics-IQ-benchmark

Artifacts in 0112, 0113, 0114 verified videos

Open
#73 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
343
Forks
47
Avg merge
5h 11m
Merged PRs (30d)
4

Description

It seems that for the three viewpoints (0112, 0113, 0114) in the "match blows balloon" scenario the artifact removal step for the "verified" set may have introduced other artifacts (see screenshot of the 'testing' split of 0014). Specifically, a light-colored box around scene probably intended to edit the video to stop the rotating platform in the 'deterministic artifacts' case. I do not see these artifacts for 0057 or 0051 with a similar rotating platform.

- Can you confirm that the artifacts I am seeing in these videos exist in the dataset and it is not a download error on my end? I re-downloaded the set multiple times and still see the artifacts. I do not see the artifacts in the original physics IQ videos.
- If these artifacts do exist in the dataset, would you be able to release a corrected version for these videos? I understand the importance of keeping the physics IQ verified evaluations stable, but I have other use cases for the dataset where having naturalistic videos (no artifacts like this) is important.

Thank you for this dataset and for your help!

Image

Contributor guide

Open the contributing guide

Research direction

Inspect the verified videos for viewpoints 0112, 0113, and 0114 in the “match blows balloon” scenario, especially the testing split, and compare them with the original physics IQ videos and similar rotating-platform videos 0057 and 0051. Confirm whether the light-colored box is present in the dataset rather than caused by download corruption; done means documenting the finding and determining whether corrected videos can be released.

Written by the indexing model from the issue text.

Assessment

Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.