haskell / haskell/cabal

Cabal Test 2.0

Open
#3,578 3 comments 2 reactions 0 assignees View on GitHub
Cabal: cmd/test Cabal: file format Cabal: other
Dominant language
Haskell
Stars
1.7k
Forks
750
Avg merge
4d 3h
Merged PRs (30d)
28

Description

I would like to revise the test interface in Cabal 2.0 to reflect what we have learned in the last six years.

In no particular order, here are some lessons I've picked up:
1. The standard I/O interface is good enough for most people. In no small part, this is due to the proliferation of good tools from other frameworks for dealing with this style of test.
2. The detailed interface is over-engineered. Few people need the few additional features it provides. The interface is hard to use; the cost of implementing it is far too high.
3. It is not as important as we thought for Cabal to handle logging. Many of our users need only direct-to-terminal output. (Again, this facilitates interaction with other tools, like continuous integration systems.) For the remaining users, simple log-to-file suffices.
4. It is important to provide some facility for dynamically generating tests. Integration with `doctest` is much more difficult than it should be. This should not require a custom Setup.hs!

I propose to design a new test interface to become the default interface in Cabal 2.0. In this document, I will refer to it as "test interface 2.0". I want to start by building off our success with the standard I/O interface; therefore, a "test suite" will consist of an executable which Cabal will compile. Test interface 2.0 will define a standard command-line interface for test suites; this allows us to incorporate desirable features (mainly introspection) with minimal overhead for users.

There is discussion in #3545 regarding pre-processors; it would be natural to treat test generators like `doctest` as pre-processors.

The test logging features should be removed from the Cabal library. We can provide minimal logging features in cabal-install.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.