apache / apache/nuttx

[NXDART] NuttX Distributed Automation for Build and Runtime Testing.

Open
#15,730 45 comments 2 reactions 2 assignees Claimed by @cederom View on GitHub
Area: Testing Type: Enhancement
Dominant language
C
Stars
4k
Forks
1.7k
Avg merge
1d 17h
Merged PRs (30d)
237

Description

* In order to maintain high quality of the code releases we need not only Build but also Runtime Automated Testing on a Real World Hardware Boards.
* GitHub CI provides automated Pull Request verification (@lupyuen manages). TestBot already operational and can be controller with comment commands.
* We need to test built code on a physical devices in order to validate drivers etc.
* Solution should be distributed, so anyone can test what they have at hand/home/work/desk.
* It's called Distributed Test Environment because it enables independent decentralized cross-check examination.
* Distributed design provides availability redundancy.
* Solution is designed to be zero-cost and low maintenance cost. Tested already on rPI-Zero-2W that takes 1W idle and 5W build power with ~400 sec build time of medium size configuration (`raspberrypi-pico:nsh`).
* Some sort of central Dashboard is required for build/runtime logs storage, analysis, reporting (@lupyuen manages). NuttX Dashboard already updated and serves TestBot to do runtime on real world hardware boards.
* It will objectively indicate which change broke (or is going to break) what and where, so project can react accordingly.
* Use case: Some old PC as local CI machine, attach available board(s), build and runtime test master, report problems to Dashboard.
* For now we are working on existing board configurations (`nsh` and `ostest`) and adapt tests scenarios. Board configs need cleanup and standardizatiion (i.e. `:nsh` only contains `nsh` nothing else, and standard set of built-ins`.
* All tests must return `PASS`, `FAIL`, `TIMEOUT`, `ERROR`, `SKIP`, `UNAVAILABLE` results for every single test case plus detailed logs.
* PASS - test results as expected.
* FAIL - test result not as expected.
* ERROR - test execution failed, for any reason, test result is unknown.
* TIMEOUT - operation did not finish in available time limits.
* UNAVAILABLE - test is unavailable due to hardware restrictions.
* SKIP - test was not performed, but on a test list.
* Tests should define `ASSERT` / `EXPECT` conditions, so tests that are designed to fail are possible too (i.e. invalid parameter).
* Future: List of external and internal tests needs to be created for all boards, and boards/configuration specific tests. This can be implemented as `selftest` board configuration. Every board should at least provide mandatory base `selftest`, and optional `selftest-extended`, `selftest-specific`, `selftest-custom`, etc (see below). This is a song of future.
* Future: We need a building blocks for base / extended / specific / custom test scenarios. Base tests will be mandatory and cover all boards (i.e. nsh + help + ostest). Extended may include benchmarks, stress tests, will be optional (may not be possible on small platforms) but may provide additional results like performance improvement or degradation. Specific tests will be optional too and would cover arch / board specific tests. There must be a way to implement Custom scenarios for closed testing of custom hardware etc.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.