[NXDART] Test Scenarios.
- Dominant language
- C
- Stars
- 4k
- Forks
- 1.7k
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 237
Description
* In order to provide generic approach and universal architecture we need to define a standard set of test cases.
* There are two main categories of tests: Build and Runtime.
* Runtime tests can be: Internal (working as the NuttX binary application "inside" the MCU, no external components required, even onboard), and External (NuttX interaction with external world outside MCU takes place, i.e. sensors, network, WIFI, BLE, GPIO, other interfaces and peripherals).
## Build Tests
* Mandatory: Build current `nuttx` and `nuttx-apps` master. This ensures current HEAD quality.
* Optional(?): Build current `nuttx` with older `nuttx-apps` (i.e. release package/tag or just N-steps back in repo history). This ensures backward compatibility between `nuttx` and `nuttx-apps`.
* Mandatory: Cover each architecture (MCUs to be selected).
* Optional: Cover each possible arch/board.
* Others?
## Runtime Tests
* For now we use at least `:nsh` and `:ostest` runtime tests on existing configurations. `nsh` is the quickest way to build and verify board bringups. `ostest` validates various NuttX mechanisms (and will/should be extended with more).
* TODO: `:nsh` needs standardization and cleanup on all boards, so it only contains `nsh` and no other applications (build time), and exactly the same set of built-in features (cohoerence).
In future when basics are finished, test groups may be created according to concepts presented below.
### Runtime Tests: Base
* These tests are mandatory.
* Must be implemented and supported on every board under evaluation.
* Ensures quality of builds provided by Build Tests.
* Test Cases (to be defined):
* `nsh`.
* `uname -a`.
* `help`.
* `ostest`.
* `free`.
### Runtime Tests: Extended
* These tests are optional.
* Ensures performance validation (stability/improvement/degradation).
* Test Cases (to be defined):
* stress tests.
* benchmarks.
### Runtime Tests: Specific
* These tests are optional, they may be hardware specific (thus name).
* Ensures architecture / board / peripheral validation.
* If generic tests are used on different boards with no hardware support then `UNAVAILABLE` should be returned rather than `FAILED`.
* Test Cases (to be defined):
* Ethernet.
* WIFI.
* Bluetooth.
* LoRa.
* LCD.
* Touchscreen.
* Joystick.
### Runtime Tests: Custom
* Vendors must be able to re-use our framework on-premises with custom hardware to perform their own tests in a generic way.
* These tests can also reveal problems with our code (i.e. compatibility, efficiency, etc)
Comments below are welcome :-)
Contributor guide
Assessment
This issue has not been assessed yet.