denoland / denoland/std

testing/bdd.ts with flat test grouping style the only option scoped to top level test instead of the file

Open
#2,096 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
3.6k
Forks
681
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

feature: https://github.com/denoland/deno_std/pull/2067

The issue with this feature is that `Deno.test` is called by the `describe` function. If tests are added to the group after the `describe` function call completes, it can handle that, except for the only option. Because `Deno.test` was already called without the `only` option, it cannot retroactively change the test to have the `only` option set to true if one of the tests or sub groups are focused using `describe.only` or `it.only`.

Issue 1: When using flat test grouping, if the top level test grouping is not focused but a child is, other top level tests that are not focused will still run. The test or tests that are focused within the top level describe will just be the only tests that run within the context of that top level test grouping.

Issue 2: When using flat test grouping, If the top level test grouping is not focused but a child is along with another top level test. Only the focused top level tests and test groups will run.

**Steps to Reproduce**

Issue 1: getAge is focused but the Example A and Example B test cases still run. That's because the "User" test was registered with `Deno.test` without the `only` option.

```ts
import {
assert,
assertEquals,
assertStrictEquals,
assertThrows,
} from "../asserts.ts";
import { describe, it } from "../bdd.ts";
import { User } from "./user.ts";

it("Example A", () => assert(true));

const userTests = describe("User");

it(userTests, "users initially empty", () => {
assertEquals(User.users.size, 0);
});

it(userTests, "constructor", () => {
try {
const user = new User("Kyle");
assertEquals(user.name, "Kyle");
assertStrictEquals(User.users.get("Kyle"), user);
} finally {
User.users.clear();
}
});

const ageTests = describe({
name: "age",
suite: userTests,
beforeEach(this: { user: User }) {
this.user = new User("Kyle");
},
afterEach() {
User.users.clear();
},
});

it.only(ageTests, "getAge", function () {
const { user } = this;
assertThrows(() => user.getAge(), Error, "Age unknown");
user.age = 18;
assertEquals(user.getAge(), 18);
});

it(ageTests, "setAge", function () {
const { user } = this;
user.setAge(18);
assertEquals(user.getAge(), 18);
});

it("Example B", () => assert(true));
```

```
kyle@kyle-XPS-15-9560:~/Projects/deno/deno_std$ deno test testing/bdd_examples/user_flat_only_issue_1_test.ts
running 3 tests from file:///home/kyle/Projects/deno/deno_std/testing/bdd_examples/user_flat_only_issue_1_test.ts
test Example A ... ok (6ms)
test User ...
test age ...
test getAge ... ok (5ms)
ok (9ms)
ok (13ms)
test Example B ... ok (3ms)

test result: ok. 3 passed (2 steps); 0 failed; 0 ignored; 0 measured; 0 filtered out (42ms)
```

Issue 2: "getAge" and "Example B" are focused but only the "Example B" test cases runs. That's because the "User" test was registered with `Deno.test` without the `only` option and the "Example B" test case was registered with `Deno.test` with the `only` option. The following example is the same as the one from issue 1 but with the "Example B" test case focused.

```ts
import {
assert,
assertEquals,
assertStrictEquals,
assertThrows,
} from "../asserts.ts";
import { describe, it } from "../bdd.ts";
import { User } from "./user.ts";

it("Example A", () => assert(true));

const userTests = describe("User");

it(userTests, "users initially empty", () => {
assertEquals(User.users.size, 0);
});

it(userTests, "constructor", () => {
try {
const user = new User("Kyle");
assertEquals(user.name, "Kyle");
assertStrictEquals(User.users.get("Kyle"), user);
} finally {
User.users.clear();
}
});

const ageTests = describe({
name: "age",
suite: userTests,
beforeEach(this: { user: User }) {
this.user = new User("Kyle");
},
afterEach() {
User.users.clear();
},
});

it.only(ageTests, "getAge", function () {
const { user } = this;
assertThrows(() => user.getAge(), Error, "Age unknown");
user.age = 18;
assertEquals(user.getAge(), 18);
});

it(ageTests, "setAge", function () {
const { user } = this;
user.setAge(18);
assertEquals(user.getAge(), 18);
});

it.only("Example B", () => assert(true));
```

```
kyle@kyle-XPS-15-9560:~/Projects/deno/deno_std$ deno test testing/bdd_examples/user_flat_only_issue_2_test.ts
running 1 test from file:///home/kyle/Projects/deno/deno_std/testing/bdd_examples/user_flat_only_issue_2_test.ts
test Example B ... ok (5ms)

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out (25ms)

error: Test failed because the "only" option was used
```

**Expected behavior**

In the first issue example, the test titled "getAge" should be the only one that runs. The "Example A" and "Example B" test cases should not be running.

In the second issue example, both the test titled "getAge" and the test titled "Example B" should run. The test titled "getAge" is currently not running.

**Environment**

- OS: Ubuntu 20.04
- deno version: canary
- std version: 0.135.0? The change to add testing/bdd.ts has been merged but a std release has not been made for it yet.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.