apache / apache/nuttx

[FEATURE] Migrate CI test cases to NTFC

Open
#17,717 2 comments 2 reactions 2 assignees Claimed by @raiden00pl View on GitHub
Type: Enhancement
Dominant language
C
Stars
4k
Forks
1.7k
Avg merge
1d 17h
Merged PRs (30d)
237

Description

NTFC is a new testing tool based on test cases for NuttX from Xiaomi that allows you to automate NuttX tests for various targets (sim, qemu, hw with serial port). All features of currently used in CI tool (`tools/ci/testrun`) are supported (and much more).

Here is a discussion on mailing list: https://lists.apache.org/thread/618konqxrl38oq7049njgks2y0x5mllq

At the moment, the repositories with NTFC and test cases are on my Github account:

- https://github.com/szafonimateusz-mi/nuttx-ntfc
- https://github.com/szafonimateusz-mi/nuttx-testing

They will eventually go to the Apache Foundation account if the proposal is accepted, but at the moment it's easier for me to develop the project in a private repo. Once NTFC is running stably on NuttX CI, I think it will be a good time to move the repositories to Apache.

### **Roadmap**

Initial plan to add NTFC support in NuttX CI:

- [x] Modify nuttx and nuttx-apps repo to support NTFC.

https://github.com/apache/nuttx-apps/pull/3291
https://github.com/apache/nuttx/pull/17728

- [x] Migrate `sim/citest` to NTFC and observe if everything works.

https://github.com/apache/nuttx/pull/17728

- [x] Migrating the rest of `citest`:
https://github.com/apache/nuttx/pull/17766

- [x] Add support for `citest` for x86_64
need qemu-system-x86_64 package in docker image

- [x] Add support for `citest` for arm64

- [ ] Enabling more test cases for `citest` targets
The tool automatically detects which tests can be run based on the provided ELF. We just have to enable the appropriate applications in defconfig.

- [X] Migrate nuttx-ntfc and nuttx-testing repositories to Apache repositories.
https://github.com/apache/nuttx-testing/pull/102
https://github.com/apache/nuttx-ntfc/pull/1
https://github.com/apache/nuttx-ntfc-testing/pull/1

- [X] Official release of NTFC as Python package.
https://pypi.org/project/ntfc/

- [ ] Add stable NTFC package to the Docker image.

- [x] Remove legacy `tools/ci/testrun`.

- [ ] (SOMEDAY) Run test cases depending on GitHub labels. Running all tests in most cases makes no sense. With NTFC, we can ignore individual test modules and save CI resources.

- [ ] (SOMEDAY) Network tests on CI. We can set up a virtual network (ETH, CAN) in which several nuttx instances communicate with each other and NTFC runs tests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.