Idea: automatically test examples

Pre-scriptum: sorry for the long text

It is possible to add Rust code into documentation (equivalent of scaladoc). Not only it supports syntax highlighting, it also enable testing the examples, to make sure it is up to date with the documented API.

An idiomatic way to write an example in Rust is to call a function, then assert its result, so in the example there is the assertion into the which the user/reader sees the returned value. For things not returning a value (we have plenty of them to describe hardware), no assertions is done (there are already unit tests elsewhere in Rust, just like with Scalatest), and it still checks that it builds and does not panic (~ throw an exception).

For SpinalHDL, it could be possible to check if examples build and if it returns an error or not (for instance design errors). For full examples with hardware + test, check if the test is successful.

I think the main question is: how have a source translated to RTD and scala test files?

First, to check with current API I think it is possible to clone the SpinalHDL template for tests and add / replace files (GitHub actions or other mechanism would enable caching dependencies).

Then, how to have testable code? It does not seem possible to automatically translate all current examples and voilà the tests run and pass. I see several ways, all of them are opt-in and would not break current documentation (so the documentation would need refactoring of each test to make it testable). These are listed below (to be updated during discussion):

Tests in scala files

Sphinx can include files as code blocks, and these files can be used in a scala environment but it does not seem easy to deal with:

• Sphinx can take them as-is
• How to integrate them into tests?
• You may not want to show all the lines to not have heavy full examples everywhere in the docs. When showing code which goes into a component, adding a "component" start and end all the time may be noisy.
• Put all examples in the same directory and have paths like ../../examples/section/subsection/example.rst? Or put examples with the code and make them harder to find to execute tests?

Partial tests in scala files

Having Scala files with only shown part of a test and some metadata to help including it in a test. Still needs to be injected in some code.

Parsing rst files

Please don't do that.

Writing a Sphinx extension

I think it is the most flexible solution so I'm gonna write a little bit about what I think we could get.

This extension, written in Python, would bring tags like .. spinalComponent, followed some metadata, then by code (like current situation). It would generate an example like:

test("some test name from some metadata?") {
  class ComponentNameFromMetadata extends Component {
    the code of the example
  }
  SpinalConfig.generateVerilog(new ComponentNameFromMetadata)
  // I would like a "check" function to not generate a file ^^'
}

There would be several tags, for several contexts (Component, full test, maybe others?).

It would also make it possible to have more features:

• Hiding some lines (for instance declaring signals, but the example is often more clear if they appear on it) by prefixing them with a # (like Rustdoc)
• Checking that some code does not build (scalatest has a function for that) or has design errors (check that generation throws an error), and in this case print the example a different way.
• A "copy" button to copy example code wrapped into an App to play around with it, or a button to show this App.

The issue names a proposed Python Sphinx extension, Scala test files, RTD output, GitHub Actions, and a SpinalHDL test template, but no repository files. Start by reviewing the current documentation examples and test template, then determine how opt-in examples should be represented. Done means examples can be checked for builds, errors, or passing full tests without breaking existing documentation.