Enhance and automate acceptance tests to accelerate releases

I usually run the acceptance test suite documented in https://github.com/actions-runner-controller/actions-runner-controller#contributing plus random manual test cases I have come up with at the time of cutting the new release.

The process is time-consuming and error-prone.

And that makes people frustrated and asking for a new release(like #562), further delaying the next release by consuming my time for responding. Even after finally cutting the new release, there are still unrecognized problems like #427, #467, and #468 which confused users.

Can we do anything better?

My own idea is not that good- Just somehow enhance and automate acceptance tests. Ideally, it should run on every commit and every pull request. It should also cover more cases, like #560.

Automation isn't that easy.

First, which test environment should we use? Should we use kind for the acceptance test environment? Or anyone could contribute some infrastructure for running a real k8s cluster for acceptance testing?

Second, how can we enhance the test suite in a sustainable manner? What we have today is just a set of shell scripts and template files https://github.com/actions-runner-controller/actions-runner-controller/tree/master/acceptance. Does it scale when we add more test cases and data?

Third, how can we run github actions workflows in tests? We have some workflow definitions for testing. But our acceptance test doesn't run it. We need to set up some github repository and organization for testing, deploy runners associated to the repo and the org, push a git commit to a repo to trigger a workflow run for testing. How can we reliably automate that process?

There might be more. If you're willing to help on any of the problems listed in this issue, or have comments, please reach us 😄

Start with the acceptance-test instructions in CONTRIBUTING and inspect the shell scripts and template files under acceptance/, including the workflow definitions in acceptance/pipelines. Compare the current coverage with cases such as #560, then determine how tests could run on commits and pull requests and reliably exercise GitHub Actions workflows using a test repository and organization.