Lower the entry barrier for testing pull requests
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 17.4k
- Forks
- 3k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 14
Description
Motivation
Due to the lack of an effective automated testing suite (and the complexitiy to build such tooling), OpenRA requires repetitive ingame testing for nearly every pull request. While testing itself doesn't require any technical skills, the requirements of getting the pull request branch to work locally are huge for somebody who has never had any contact with software development or managing OS on a technical level. So there is likely a group of people out there that could help with testing ingame changes and contribute in a valueable way if the entry barrier was lowered to basically "download something, doubleclick it, test what is asked for, report your findings".
Proposed solution
1.) Package "installers" for pull requests automatically and make them available for potential testers
GitHub actions can be used to build and deploy artifacts like binaries and more to Github Storage. This will not clutter the release area, instead they will be listed inside the "Actions" Tab but can also be accessed via the GitHub actions API. So there should be a dedicated static webpage that retrieves the list of artifacts with links to the individual installer files.
- https://docs.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts
- https://github.com/actions/upload-artifact
- https://developer.github.com/v3/actions/artifacts/#artifacts
2.) Organize the pull requests that can be tested effectively ingame
Ideally, the overview of the pull requests artifacts should only list such PRs, that can be effectively tested and are ready for testing. I don't know how this could be managed, perhaps it is possible to triggger the deploy action only when some speical condition is met, like if a certain label is present or something like this. Perhaps @orabot could learn to trigger the build. Another requirement would be to remove an artifact when the pull request is either closed or merged.
3.) Provide an easy way for testers to know what to test
This information should include the following:
- what is the change that needs to be tested and what is the expected behavior
- which mechanics or areas to look for regressions
- in which mods does this need to be tested
- TESTCASE if nessecary
This information could of course be part of the pull request description or be provided as some sort ofTEST-INFOwhich is submitted along with the PR code changes, included in the deployed artifacts and used as a description in the overview page with pullrequest installers
In this case, this should also contain some general information about the reviewing process, where to report findings, etc. It would need be removed before merging (in some automated way...?).
4.) Advertise the new testing workflow
... and hope that it motivates people to contribute. Perhaps add additional motivation by assiging discord roles or things like this.
Side effects
Possibly more "noise" from people who might start looking at the details for the first time, findings unrelated to the pull request, possible overhead due to moderation and support.
Overhead of documenting the TEST-INFO for things that are ridiculously obvious for the author and other developers. Objection of developers to provide this information and waste their time on this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository files or tests are named. Start by reading the linked GitHub Actions artifact documentation and examining how OpenRA's workflows currently build pull requests; define the artifact, readiness, cleanup, tester-information, and reporting requirements before implementation. Done means a documented, maintainable workflow that lowers the testing barrier without relying on unspecified project changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100