Reordering should not cause a module's tests to be split before/after other modules

Please bear with me a moment for an explanation - there's a good chance you won't agree with me initially, and I have a sliding scale of thoughts on what could be done with my experience.

To start, as demonstration, suppose I have a suite that acts like the following:

var globalFlag = false;
QUnit.module('A: should never fail anything, but can cleanup global things', {
    after : function () {
        globalFlag = true;
    }
});
QUnit.test('always passes', function (assert) {
    assert.ok(true);
});

QUnit.module('B: should I have one fail, or two?');
QUnit.test('fail if globalFlag is true', function (assert) {
    assert.notOk(globalFlag);
});
QUnit.test('fail if globalFlag is true, or just because', function (assert) {
    assert.notOk(globalFlag);
    assert.ok(false);
});

After running the tests in my browser, and refreshing once or twice, I see the following pattern; on one refresh, 1 failure, on my next refresh, 2 failures:

"Module interrupted" scenario - two failures

B's "before"
*** B: should I have one fail, or two? *** Test : fail if globalFlag is true, or just because
*** A: should never fail anything, but can cleanup global things *** Test : always passes
*** B: should I have one fail, or two? *** Test : fail if globalFlag is true
B's "after"

"Module uninterrupted" scenario - one failure

B's "before"
*** B: should I have one fail, or two? *** Test : fail if globalFlag is true
*** B: should I have one fail, or two? *** Test : fail if globalFlag is true, or just because
B's "after"
*** A: should never fail anything, but can cleanup global things *** Test : always passes

So, why do I consider this an issue?

I accidentally got myself into a situation that acted like the above yesterday. In practice, I was unit testing a jQuery widget, and giving the QUnit 2 - style API with before/after a try for the first time. Historically, our tests using the jQ widget create a widget on #qunit-fixture in beforeEach, and tear it down in afterEach.

This is a little inefficient, taking some 20-50milliseconds, so I wanted to try out a new way. I wanted to create the widget in before, reuse it for each test by performing much cheaper clearAll-type methods in afterEach, then finally tear it down in after.

I was successful in this, but then as I was putting together a new test, before the test was good, it failed all the time. I was very confused when more than just this test failed, though. It was because other modules were jutting in, trying to create a widget on #qunit-fixture, actually getting the existing instance from the before of my new module (since after hadn't run yet, since module B hadn't finished all its tests yet), and tearing down the grid that module B had created. Since module B only created the widget in before, subsequent tests from module B always failed.

What do I propose?

My guiding instinct here is that before/after are not all that useful if there's no guarantee that a module will run all of its cases together; if they can get interrupted by unrelated tests, people can burn a few hours trying to figure out why. In my case, this was a jQuery widget, but I could see similar circumstances for sinon spy/stub injection on prototypes, dependency injection swaps, etc.

So, I propose one of the following, in order of personal preference:

• If QUnit.config.reorder is set to true, randomize the order that modules run in, and the order that tests run in within a module if desired, but do not reorder such that a module B's tests can have module A's tests run in between B's before and after hooks
  • Correct me if I'm wrong, but I feel this is what JUnit does for Java. Perhaps I haven't looked carefully enough
• If QUnit.config.reorder is set to true, ensure that any module that defines before/after hooks in some fashion cannot get interrupted by tests from another module (but leave legacy behavior intact for modules w/o before/after)
• Add a new configuration option, defaulted to whatever, that if toggled on would enable one of the two above behaviors
• Update the documentation to indicate that modules can be interrupted in this fashion, linking to that explanation from the docs for before, after, and QUnit.config.reorder

Overall

I do like QUnit, I just wonder if folks other than me might run into this same situation. But anyway, thank you for reading this far!

Start by reproducing the described QUnit.config.reorder scenarios with modules that use before and after hooks, and compare interrupted versus uninterrupted execution. Determine which proposed behavior should be implemented, then define completion around preserving module hook boundaries or documenting the interruption behavior, since the issue presents several alternatives without selecting one.