rr-debugger / rr-debugger/rr

Replay gets stuck or diverges on a Firefox plugin-container for Widevine protected video

Open
#3,246 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

STR:

  1. Recording Firefox (I've tested on local builds of mozilla-central but an official build will probably work), go to https://shaka-player-demo.appspot.com/demo/, install the plugin if prompted, and play “Sintel” (should be marked as Widevine-protected); once it starts playing, the browser can be closed.
  2. Replay the last plugin-container process (should have gmplugin as the last arg and mention gmp-widevinecdm); when the gdb prompt appears, c. (If the sandbox isn't disabled (see below) and this isn't being run from a Firefox source tree, do handle SIGSYS nostop noprint first.)

In one case this gave a Replay diverged; target registers at ticks target mismatched error; in other cases, it got stuck with an rr:plugin-container process spinning on the CPU in userspace, with the main rr process blocked in waitid waiting for it to stop, which it never did.

I first encountered this on an AMD Threadripper PRO 3975WX but have also reproduced it on an Intel Core i9-9900K, so I don't think it's CPU-specific. The kernels are both Debian 5.17 versions. The sandbox can be disabled (MOZ_DISABLE_GMP_SANDBOX=1 in the environment) or not; it doesn't seem to make a difference.

At one point @khuey suggested uploading the trace to Pernosco, which was able to replay the whole thing. There was also an attempt to use rr rerun on the part of the trace where the replay got stuck, which resulted in this error:

[FATAL src/PerfCounters.cc:733:read_ticks()] 
 (task 225329 (rec:214287) at time 3524)
 -> Assertion `!counting_period || interrupt_val <= adjusted_counting_period' failed to hold. Detected 562916 ticks, expected no more than 550033

And the divergence error I got on that one replay attempt that didn't just silently get stuck:

(rr) c
Continuing.
[ERROR src/ReplaySession.cc:775:guard_overshoot()] Replay diverged; target registers at ticks target mismatched: 
[FATAL src/ReplaySession.cc:784:guard_overshoot()] 
 (task 323030 (rec:320100) at time 489978)
 -> Assertion `false' failed to hold. overshot target ticks=105440383 by 2

(Also, just for the record: I am specifically not trying to inspect the DRM module itself; I was debugging an issue in the Gecko code running in that process.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Firefox plugin-container replay using the provided STR, then inspect the failure locations named in src/PerfCounters.cc:733 and src/ReplaySession.cc:775-784. Compare the divergent and stuck cases, including rr rerun, and consider the issue addressed when the trace replays without divergence or an endless userspace spin.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.