ocaml / ocaml/dune

Improve testing facilities / ability to run an individual test

Open
#6,142 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

test
Dominant language
OCaml
Stars
1.9k
Forks
500
Avg merge
15h 21m
Merged PRs (30d)
277

Description

Apologies if this is a duplicate, I haven't been able to find a similar request.

I have seen in the doc that the currently recommended way to run a test is to dune exec its corresponding executable. However, this assumes that all tests are binaries, most likely testing API/library functions.

In our case, dune builds a complex script interpreter (liquidsoap) that also relies on having access to a standard library of scripted values from the scripting language. Tests for the interpreter run the interpreter with specific scripted tests.

What would be ideal for us would be to:

  • Be able to have a simple easy setup of the test environment. Currently, we rely on alias and rule dependencies to setup all that is needed. This quickly ends up being pretty verbose and is also obscure, some times changing slight dependencies means that e.g. the standard library is no longer accessible. Ideally, we should be able to pin specific assets build by dune that need to be available in the testing environment
  • Be able to run individual tests. We can narrow down tests that are ran by placing them in a folders but, when debugging, it would be easier to be able to run a single faulty test. I have tried multiple ways to make this work, mainly using rule dependencies or building a ad-hoc test runner that would be a compiled exe wrapping over a process create call. However, these end up being pretty hairy due to the issues with setting up proper test environments as stated above.
  • A way to run over multiple tests without having to specify a rule entry for each of them. Currently, we generate custom dune.inc files to include and have to re-generate them each time a test is added/removed. This is prone to issues with the dune files not being in sync with the test files and is also pretty hard to read once auto-generated.

Thanks for the hard work on dune. Overall, we are super happy with it otherwise!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the documented dune exec workflow and the alias and rule dependencies described in the issue. Review the generated dune.inc approach and existing test entry points. Done means a designed and documented way to set up test assets, select an individual test, and run multiple tests without one rule entry per test.

Written by the indexing model from the issue text.

Assessment

Tech stack
ocaml
Domain
build-system, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.