lambdaisland / lambdaisland/kaocha

Make a distinction between load errors in the test file being loaded vs. in other namespaces.

Open
#27 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

ergonomics improvement
Dominant language
Clojure
Stars
860
Forks
81
PR merge metrics
No merged PRs in 30d

Description

If an exception occurs while Kaocha loads a test namespace, then a warning is printed, the :kaocha.type/namespace test is marked as failed, and loading/running continues.

This is so that when you have a typo which prevents a test namespace from loading, this doesn't stop the world. Instead it just runs the rest of the tests, and reports a failure pointing at the syntax error.

The problem with this approach is that loading a namespace could fail because a dependent namespace has issues. In the degenerate case a source namespace that is used indirectly in every test fails to load, causing a failure to be reported for every test namespace. This is especially easy to do (and annoying to encounter) in --watch mode.

Instead when loading a test namespace we should first evaluate its ns form. If that causes issues then we hard abort the test run. Only after the ns form has loaded do we load the full namespace, and turn load errors into test failures.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Kaocha's test-namespace loading path and the :kaocha.type/namespace test behavior. Check how the namespace's ns form is evaluated separately from the remaining forms, then verify that dependency load errors abort while test-namespace load errors become failures. Done means the distinction works in normal and --watch runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.