lambdaisland / lambdaisland/kaocha

Change randomize? at any level

Open
#272 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

improvement
Dominant language
Clojure
Stars
860
Forks
81
PR merge metrics
No merged PRs in 30d

Description

Problem

The majority of the time, I like running my tests with randomize? true. Rarely, I will have a case where running tests for a specific test suite or namespace would be beneficial. Typically, this is because an earlier test(s) do an expensive, mutative operation that I do not want to redo. This might feel close to a :once fixture. However, there can be an important difference -- the expensive operation is the thing we actually want to test. As such, a :once fixture does not feel like a good fit.

Proposal

I propose adding support for setting :kaocha.plugin.randomize/randomize? on a test suite map and namespace.

Suggested implementation path

See relevant code. If the test-plan has :kaocha.plugin.randomize/randomize? set as a top-level key or in the [:kaocha.testable/meta :kaocha.plugin.randomize/randomize?] path, do not sort the :kaocha.test-plan/tests, but still continue the recursion, allowing the toggle to be on/off at any level.

If the above sounds good, I can provide a PR.

Alternatives

  1. A harsh alternative is to set :kaocha.plugin.randomize/randomize? false directly in tests.edn for all tests that use that config. This is not ideal because most often you want randomize? true.
  2. Always pass --no-randomize to the CLI when running the tests. This is not ideal because it requires the caller to know they must pass that flag for tests to be idempotent.
  3. Execute the expensive operation multiple times. This is, perhaps, the cleanest path, but costs us (potentially significantly) extra time when running tests locally and on CI.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/kaocha/plugin/randomize.clj at the referenced randomization logic and trace how the test plan is recursively processed. Support the toggle at the top-level and testable metadata levels while continuing recursion, so randomization can be enabled or disabled independently for suites and namespaces. Done means each level’s setting controls whether its tests are sorted.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.