lambdaisland / lambdaisland/kaocha

Support filtering or selecting tests by filename

Open
#323 3 comments 1 reaction 1 assignee View on GitHub

@plexus is already working on this.

Since Jan 20, 2023.

Dominant language
Clojure
Stars
860
Forks
81
PR merge metrics
No merged PRs in 30d

Description

The use case

CircleCI has functionality for getting a list of test names and then distributing them across parallel test runners. It's relatively easy to make this work with test runners in other languages, because many of them provide functionality for providing a list of test files to be interpreted and run by the runner. I'd like to be able to use this somehow.

Additionally, focusing by file would make it relatively easy to quickly run a subset of tests just based on file structure from the command line without modifying any files, which to my knowledge isn't possible.

Existing workarounds

Writing a kaocha entrypoint that bashes out to the CircleCI binary

You can see an example of it here: https://andreacrotti.github.io/2020-07-28-parallel-ci-kaocha/

This feels brittle compared to providing an arglist - it's structuring a runner around shelling out rather than providing an external interface that can be piped or arg'd in.

Writing a plugin

I've currently gone this route. I was able to write a plugin that takes --focus-files and then hacks the options of kaocha.plugin/filter to provide ns filtering indirectly. This works but is brittle because it depends on the order of plugin application.

I could also go and reimplement enough of the filter plugin's functionality to make the plugin self-contained, but that seems like a duplication of effort.

Proposed solutions

Add --focus-files as an option to the filter plugin

This could live in the filter plugin and use some of plugin's existing functionality. It seems to fit with the concepts of the other filter mechanisms.

Make fully-qualified files valid entries for :test-paths

If fully-qualified files were an option here, it would be possible to write a plugin that takes a --focus-files (or similar) argument and updates the :test-paths vector with those files. This seems like it would have an additional benefit of cutting down on the number of namespaces that are actually loaded by the framework before tests begin.

Conclusion

Let me know if any of these approaches is better than another in your eyes. I'm happy to provide a PR.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.