lambdaisland / lambdaisland/kaocha
Change randomize? at any level
Nobody has claimed this yet.
- Dominant language
- Clojure
- Stars
- 860
- Forks
- 81
- PR merge metrics
- No merged PRs in 30d
Description
Problem
The majority of the time, I like running my tests with randomize? true. Rarely, I will have a case where running tests for a specific test suite or namespace would be beneficial. Typically, this is because an earlier test(s) do an expensive, mutative operation that I do not want to redo. This might feel close to a :once fixture. However, there can be an important difference -- the expensive operation is the thing we actually want to test. As such, a :once fixture does not feel like a good fit.
Proposal
I propose adding support for setting :kaocha.plugin.randomize/randomize? on a test suite map and namespace.
Suggested implementation path
See relevant code. If the test-plan has :kaocha.plugin.randomize/randomize? set as a top-level key or in the [:kaocha.testable/meta :kaocha.plugin.randomize/randomize?] path, do not sort the :kaocha.test-plan/tests, but still continue the recursion, allowing the toggle to be on/off at any level.
If the above sounds good, I can provide a PR.
Alternatives
- A harsh alternative is to set
:kaocha.plugin.randomize/randomize? falsedirectly in tests.edn for all tests that use that config. This is not ideal because most often you wantrandomize? true. - Always pass
--no-randomizeto the CLI when running the tests. This is not ideal because it requires the caller to know they must pass that flag for tests to be idempotent. - Execute the expensive operation multiple times. This is, perhaps, the cleanest path, but costs us (potentially significantly) extra time when running tests locally and on CI.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/kaocha/plugin/randomize.clj at the referenced randomization logic and trace how the test plan is recursively processed. Support the toggle at the top-level and testable metadata levels while continuing recursion, so randomization can be enabled or disabled independently for suites and namespaces. Done means each level’s setting controls whether its tests are sorted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100