chocolatey / chocolatey/choco

Overhaul how/when integration tests are ran against the code base

Open
#3,575 0 comments 0 reactions 0 assignees View on GitHub
0 - Backlog Build Automation NO RELEASE NOTES
Dominant language
C#
Stars
11.5k
Forks
960
PR merge metrics
No merged PRs in 30d

Description

### Checklist

- [x] I have verified this is the correct repository for opening this issue.
- [x] I have verified no other issues exist related to my request.

### Is Your Feature Request Related To A Problem? Please describe.

Currently, the Chocolatey CLI Integration Tests are not run as part of a typical CI build. Only the Unit Tests are executed during the CI builds. As a result, it is possible that issues are only found some time later, once the scheduled integration builds are ran.

The reason that this is done is simply because the Integration Tests are known to take longer to complete, so rather than run them on each build, they are only ran on a scheduled basis. 9 times out of 10, this isn't a problem, but on that one occassion where something would have been found earlier, we are having to wait until the results of the scheduled build completes.

In addition, it is not currently possible to manually trigger the scheduled integration build, as it is not possible to select all the possible branches that we could want.

### Describe The Solution. Why is it needed?

There are a couple of ways that this could get better...

1. Change all CI builds to _always_ do both unit and integration tests
2. Change all CI builds to run on each PR build
3. Change all CI builds to allow manual triggering, including the ability to select PR builds

Of the presented solution, the last one makes the most sense, as this would allow fine-grained control over _when_ a build is ran that includes integration tests. That way, if there are any concerns, for example, changes that are coming in with a PR, all the tests can selectively be ran, and if they all pass, the merge can happen. This eliminates the need to wait for the scheduled CI builds to run.

### Additional Context

N/A

### Related Issues

- Internal 8689nmppu

Contributor guide

Open the contributing guide

Research direction

Start by locating the CI build definitions and the scheduled integration-test job, then review how branches and pull requests are currently selected. Done means the integration build can be manually triggered for the relevant PR or branch and its results are available before merging.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd, cli, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.