Issues on videos with imprecise metadata
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 232
- PR merge metrics
- No merged PRs in 30d
Description
I noticed a failing unit test working on a PR. It was asserting a _wrong_ expected result (incorrectly returned by older versions of decord), that it was fixed (by accident?) by a later version.
```python
def test_video_reader_len():
vr = _get_default_test_video() # "flipping_a_pancake.mkv"
assert len(vr) == 311 #number of frames is actually 310, and current version gives the correct result
```
1. **are these tests automatically run anywhere**, and should they? :-)
2. the failing test is a symptom of something bad going on, so I took a deeper look..
The behavior of `GetFrameCount` was changed (and "fixed", as a side effect) when `GetFramePTS` was introduced in 0.3.6.
With decord 0.3.5, on ""flipping_a_pancake.mkv", the function would fall back to an approximation based on FPS*duration (from metadata). In that video, the duration metadata seems to be imprecise (I think it's actually the duration of the audio track rather than the video stream), so the estimation is off by one, and this creates issues everywhere.
For example: , a VideoReader of that video returns 311 frames (while the video actually only has 310), where the last "extra" frame actually contains another "random" frame of the video.
From 0.3.6, it seems that `GetFrameCount` will (always?) return `frame_ts_.size()`, which is more precise and fixes the issue.
**Should perhaps the approximation of GetFrameCount be completely removed by `GetFrameCount`, making sure `frame_ts_.size()` is always used, to avoid similar issues?**
3. That is not the only issue that occurred on that video. With bad or missing duration metadata, a lot of other things are still going wrong, mainly because that metadata is used in `Seek`.
With the latest version of decord, on "flipping_a_pancake.mkv", the video stream has bad duration in the metadata (same also for the other example video, both from Kinetics I guess)... leading to _very bad_ things with anything that needs to Seek or SeekAccurate (anything that is not a sequential read of the entire video..).
For example, this is an interesting failure (still occurring with the latest version):
```python
from decord import VideoReader
vr = VideoReader("flipping_a_pancake.mkv")
frames = vr[:].asnumpy() # no Seek is involved, except Seek(0); so, correct result.
b = vr[152].asnumpy() # Seek(150) "fails", actually seeking to ts = 0
(b == frames[152]).all() # returns False
(b == frames[2]).all() # returns True
```
where the nearest keyframe is 150, but `Seek(150)` goes back to 0 due to a bad duration in the metadata (negative..). **Is there a way to be robust to this kind of issue, maybe relying on the pts in `frame_ts`, rather then crucially relying on the "duration" metadata in FrameToPTS and other functions?** But videos are messy, and the pts could have also their own issues, so I don't know if that would solve everything.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.