DynamoRIO / DynamoRIO/dynamorio

add binary package automated testing

Open
#1,995 0 comments 0 reactions 0 assignees View on GitHub
Component-Tests Hotlist-Release Type-Feature
Dominant language
C
Stars
3.2k
Forks
629
Avg merge
2d 15h
Merged PRs (30d)
31

Description

The testing we perform prior to a release (see
https://github.com/DynamoRIO/dynamorio/wiki/New-Release) consists mainly of
sanity checks running all of the samples and polished tools on both small
console apps but also on graphical apps like notepad. It would be nice to
automate that.

Xref adding graphical app testing to the pre-commit suite: #120.

This issue is about adding binary package automated testing, as opposed to
the source code building and testing of our regular test suite. The goal
here is to have a script that can be run on VM's that do not necessarily
have a development environment set up, for sanity testing on a wide range
of platforms.

Step 1 is to avoid Windows popups by making SHOW_RESULTS a runtime option
(separately but with the same name in each sample I suppose). Even if the
rest of the automation isn't finished, that alone would make testing
easier.

Step 2 is to take the current sequences of commands and turn them into a
more formal script with some automated checking of results.

Step 3 is to set up cron jobs to run in virtual machines periodically.
Perhaps when each "nightly build" is created: #1967?

We do not expect to be able to easily automate some pieces:
- .msi install and uninstall (for Dr. Memory)
- Visual Studio external tool testing (for Dr. Memory)
- admin -syswide_on testing
- cmd interactive output testing (less of an issue on recent Windows)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.