clash-lang / clash-lang/clash-compiler

Improve `clash-testsuite` CLI interface

Open
#2,012 1 comment 0 reactions 0 assignees View on GitHub
enhancement tests
Dominant language
Haskell
Stars
1.6k
Forks
169
Avg merge
15h 52m
Merged PRs (30d)
46

Description

It would nice if the CLI for `clash-testsuite` was improved to

* allow easy control over which simulators are used to run the test suite
* provide more effective error reporting when simulators are not in `PATH`

For the first point, I propose adding `--simulator` and `--no-simulator` flags for each simulator we may use in the test suite. These would allow filtering of the tests based on simulator use in a convenient manner. For example

```
$ clash-testsuite --iverilog
```

would only run tests on `verilog`, and only with the `iverilog` simulator.

```
clash-testsuite --installed --no-verilator
```

would run on every simulator on the user's system with the exception of `verilator` (if it is installed). This means where we support multiple simulators for a particular HDL we can limit the jobs to the simulators we care about. The special extra flags `--all` and `--installed` could be used to force all simulators in the test suite / every simulator installed on a user's system.

Regarding reporting: currently when a simulator is not installed and it's tests are executed each test fails with a command not found error in its `stderr`. If we check in advance whether the necessary simulators are installed, we can error early without needing to generate code with `clash`, and provide a single useful error instead of generating HDL then failing for each test, e.g.

```
$ clash-testsuite --iverilog
ERROR: `iverilog` could not be found in PATH.
Please install `iverilog` or choose another simulator.
$ echo $?
127
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the clash-testsuite CLI entry point and trace how simulators are selected, checked, and invoked. Define completion as supporting per-simulator inclusion and exclusion flags plus --all and --installed, with an early, useful PATH error and exit status 127 instead of repeated test failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
cli, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.