can't run GPU-accelerated Firefox or Firefox WebGL under rr

On 2 different machines, I've been unable to run accelerated Firefox or run any WebGL in Firefox under rr.

In particular:

• if I have a Firefox profile with user_pref("layers.acceleration.force-enabled", true) in the prefs.js file, I can't start Firefox
• if I run a WebGL test, WebGL isn't available

So far I've tracked down the underlying problem to something that happens in the glxtest process that Firefox fork()s to learn about the X server's capabilities in a process that is allowed to risk crashing.

In Firefox not running under rr, I see the output of that process as:
"VENDOR\nNVIDIA Corporation\nRENDERER\nQuadro NVS 295/PCIe/SSE2\nVERSION\n3.3.0 NVIDIA 304.131\nTFP\nTRUE\n"
whereas when running under rr, the output is:
"X error occurred in GLX probe, error_code=2, request_code=154, minor_code=3\n"

This difference leads to the above symptoms.

The latter output happens as a result of the x_error_handler function being called here:

#0  0x00007f6ee91b949c in x_error_handler(Display*, XErrorEvent*) (ev=0x7ffd1db13a50)
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/toolkit/xre/glxtest.cpp:102
#1  0x00007f6ef3e7bc4d in _XError (dpy=dpy@entry=0xbb80e0, rep=rep@entry=0xba04a0) at ../../src/XlibInt.c:1429
#2  0x00007f6ef3e78ad7 in handle_error (dpy=0xbb80e0, err=0xba04a0, in_XReply=<optimized out>) at ../../src/xcb_io.c:213
#3  0x00007f6ef3e78b95 in handle_response (dpy=dpy@entry=0xbb80e0, response=0xba04a0, in_XReply=in_XReply@entry=1) at ../../src/xcb_io.c:325
#4  0x00007f6ef3e79b50 in _XReply (dpy=dpy@entry=0xbb80e0, rep=rep@entry=0x7ffd1db13c10, extra=extra@entry=0, discard=discard@entry=1)
    at ../../src/xcb_io.c:627
#5  0x00007f6ef3e7548d in XSync (dpy=0xbb80e0, discard=0)
    at ../../src/Sync.c:44
#6  0x00007f6ee39d59e3 in  () at /usr/lib/nvidia-304-updates/libGL.so.1
#7  0x00007f6ee39c9301 in glXCreateContext ()
    at /usr/lib/nvidia-304-updates/libGL.so.1
#8  0x00007f6ee91ba52c in glxtest() ()
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/toolkit/xre/glxtest.cpp:248
#9  0x00007f6ee91ba6ef in fire_glxtest_process() ()
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/toolkit/xre/glxtest.cpp:333
#10 0x00007f6ee91b15de in XREMain::XRE_mainInit(bool*) (this=this@entry=0x7ffd1db144a8, aExitFlag=aExitFlag@entry=0x7ffd1db14430)
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/toolkit/xre/nsAppRunner.cpp:3052
#11 0x00007f6ee91b6b74 in XREMain::XRE_main(int, char**, nsXREAppData const*) (this=this@entry=0x7ffd1db144a8, argc=argc@entry=4, argv=argv@entry=0x7ffd1db15898, aAppData=aAppData@entry=0x7ffd1db146b8)
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/toolkit/xre/nsAppRunner.cpp:4424
#12 0x00007f6ee91b6f26 in XRE_main(int, char**, nsXREAppData const*, uint32_t) (argc=4, argv=0x7ffd1db15898, aAppData=0x7ffd1db146b8, aFlags=<optimized out>)
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/toolkit/xre/nsAppRunner.cpp:4535
#13 0x000000000040496c in do_main(int, char**, char**, nsIFile*) (argc=argc@entry=4, argv=argv@entry=0x7ffd1db15898, envp=envp@entry=0x7ffd1db158c0, xreDirectory=0xba14e0)
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/browser/app/nsBrowserApp.cpp:259
#14 0x0000000000404101 in main(int, char**, char**) (argc=4, argv=0x7ffd1db15898, envp=0x7ffd1db158c0)
    at /home/dbaron/builds/ssd/mozilla-central/mozilla/browser/app/nsBrowserApp.cpp:392

which I presume doesn't happen when not running rr, since I don't see the output that results from it being called.

I'm not sure what would be useful to debug next.

Note that I see the first symptom (being unabled to run accelerated Firefox) on 2 different machines, though I've only checked the WebGL symptom on my desktop (which is also where I did the debugging above).

Start in toolkit/xre/glxtest.cpp, especially x_error_handler, glxtest(), and fire_glxtest_process(), using the reported GLX error and stack trace as the starting point. Compare the GLX probe under rr with normal Firefox and determine what prevents accelerated Firefox and WebGL from working; done means both symptoms are resolved.