Proposal: Bicep Testing Framework

# Proposal - Testing in Bicep

By Emily Redmond

Note: This document represents a product proposal for the future direction of testing in Bicep. You can find an overview of existing experimental testing functionality [here](https://github.com/Azure/bicep/issues/11967). All testing functionality and future proposals are subject to change. If you have any feedback, please leave a comment below so we can track it for future discussion.

## Executive Summary
Testing provides the ability to confirm code acts as a programmer intends. With test-driven development, a well-adopted and well-loved principle in software development, a developer can be confident that the actual results of a program are in line with expected results.

The Bicep language strives to draw inspiration from the best practices of software development and make them available in Infrastructure as Code to improve the DevOps experience. Thus, we will introduce a client-side unit-testing framework to the Bicep language through two new keywords, `test` and `assert` and a new file type `<.biceptest>` that requires no Azure connection or deployment to test Bicep code.

Customer research inspired and validated this approach, as Bicep users have indicated that saving time, failing fast, and being confident with deployment changes is important to their workflow. Moreover, customers complain that long authoring cycles that end in failure or incorrectly configured resources is a pain point.

## Problem Statement
Long authoring-deployment cycles, ie. dev-test cycles, are a pain point for Bicep users. Users complain that the time for some deployments can exceed 10 minutes, nearing 30 minutes in some cases like AKS and Cosmos DB deployments. Frustratingly, these deployments may then fail halfway through or near the end of this long deployment process due to an incorrectly configured resource.

Users have no way to “fail fast” and catch errors before deployment today. Further, they have no way to validate their Bicep code offline on the client-side (ie. locally without a connection to Azure).

If a deployment completes successfully and a user realizes after the resources have been created that they used an incorrect naming convention, misconfigured certain properties, or simply made a typo, there is no way to easily change the resource settings or roll back the deployment. In many cases, the user must completely tear down or delete the created resources and return to the Bicep authoring stage, thus restarting the 10+ minute deployment cycle.

For example, imagine a Bicep dev has named a resource “${location}-${env}-service-app” and realizes after successfully deploying the resource they did not follow their organization’s expected naming convention “${env}-{location}-service-app”.

## Goals
The overarching goal of introducing testing to Bicep is to further improve the quality and maturity of the Bicep language. Goals specific to our proposed testing solution include:

- Push error detection further left in the author-deployment development cycle

- Focus on client-side testing without connecting to ARM. This tightens the dev/test loop, facilitating early validation at the Bicep-user level rather than the organization level, thus empowering individual Bicep developers to be confident in deployments.
- Support unit testing. With Bicep’s focus on modular development, it is essential that customers are confident in each module as an individual unit before combining modules and using/reusing them as building blocks for a template. Unit tests allow for validation and confidence in individual modules.

- Empower Bicep developers and their infrastructure teams to be confident in their deployments and maintain modules used across the team.
- Create an integrated Bicep testing experience that doesn’t require any additional components to be installed and easily integrates with current developer workflow.

A **non-goal** of this project is to replicate test frameworks in other established software development languages. While we want to draw inspiration from and learn from users’ perception of existing testing frameworks, we want to build a cloud infrastructure-specific approach.

## Target Personas

### Bicep DevOps Engineers

**User persona:** DevOps engineers and Cloud Architects who deploy infrastructure for app teams and broader organization at the enterprise-level, particularly teams or contractors who have multiple developers contributing to complex templates.

**Testing need:**
- Testing helps maintain modules and ensures consistency and conventions across the team. Testing empowers individual developers to be confident in deployments and thus supports the efficiency of the entire team.
- Bicep contractors can provide a set of tests to their clients with their Bicep templates, which allows their clients to be more confident in updating the templates as needed without fear of breaking changes by running their changes against the provided tests to ensure the system still works as intended.

## Definitions
**Push left** – To move testing or any component of development ‘left’ in the process diagram towards the beginning of the development cycle to save the developer time or effort and catch errors or inconsistencies as soon as possible and with less friction. “Failing fast” is related to pushing testing left

**Assert / Assertion** – Common testing framework that allows a developer to provide an expected or intended output and compare it to the actual results of code output via a Boolean expression

**Mocking** – Run-time functions and results can be mocked on the client-side (ie. Without connection to Azure), thus allowing a developer to test code components on the client-side that are usually dependent on run-time functions

Test Scopes:
- **Unit test** (Current focus for Bicep testing) - Validates the integrity of a single component without considering dependencies. Unit tests run offline on the client-side (ie. Without connection to Azure)
- **Integration test** - Runs on and between all the units that make up a functional group, e.g. infra for a single app
- **E2E test** - End to End test. Confirms the end state of a deployed app conforms to expectations

Cloud Test Phases:
- **Pre-Deployment** – Includes authoring a Bicep template and converting a Bicep template to an ARM template. This phase is completely offline, ie. no connection to ARM or Azure
- **During-Deployment** – Includes calling the Azure Resource Manager (ARM) deployments API and the connecting to the resource provider of every resource defined in a Bicep template. This phase requires a connection to ARM and each resource provider.
- **Post-Deployment** – Includes evaluating or reviewing created resources in Azure. This phase requires a connection to Azure to review existing resources.

## Solution
### Proposed Solution:

Introduce `assert` and `test` keywords to Bicep supported by a new `.biceptest` file type. The focus of introducing testing to Bicep is to provide client-side test functionality, addressing a currently unmet user need for offline, pre-deployment validation.

After developing client-side unit test blocks and corresponding `assert` statements, we will introduce mocking within `test` blocks to further the capability of client-side testing, including common Bicep functions that depend on run-time values (eg. resourceGroup.location()).

Throughout development of all testing solutions, testing should be supported in both the CLI (Command Line Interface) and in CI/CD (Continuous Integration/Continuous Deployment) pipelines.

With unit tests, Bicep users can confirm they have met (a) organization-defined conventions, such as company naming standards, and (b) Azure-defined requirements, such as Azure maximum character lengths, before deploying.

Assert statements allow users to fail fast and iterate quickly if they fail their defined tests. This reduces the author-feedback or dev-test deployment cycle, by allowing users to short circuit long deployment wait times that would end in failure or incorrectly configured resources that are difficult to edit once created in Azure. The Expected vs. Actual test output results clearly guide users’ debugging.

This solution provides value for our users by empowering individual Bicep developers to be confident in their deployments. Testing reduces friction when creating Azure resources, thus improving the Azure cloud experience.

### `test` and `assert` keywords in `.biceptest` files
- `.biceptest` is a new file type to support Bicep unit tests where the `test` and `assert` keywords are active (ie. these keywords are only functional within .biceptest files)
- `test` blocks reference individual .bicep files and pass in parameters to mock deployment conditions, thus defining a client-side unit test

```
test mainTest 'main.bicep' = {
params: {
location: 'eastus'
env: 'prod'
}
}
```

- `assert` statements are boolean expressions that can validate template conventions and expectations of a .bicep file that is referenced by a test block. Assert statements support Bicep logic and functions, such as “contains()”, “length()”, etc. Assert statements are authored within a .biceptest file and symbolically refer to values within test blocks.
- When users author assert statements in Bicep the assert statements get converted to the equivalent ARM template assert statements depending on their deployment stage (template-level, resource-level, or resource properties-level) upon building the .bicep file.

```
assert mainServiceAppName = contains(testMain.resources.appServiceApp.name, '${env}')
```

### Mocking

A common Bicep scenario is to refer to a resourceGroup() location within a Bicep template, which requires a connection to ARM to process. We can test components like these that depend on functions that require a connection to ARM by mocking them on the client-side in test blocks. We see mocking as an important tool to continue pushing testing functionality to the client side in future test iterations. See Milestone 3 for implementation details and possible syntax.

Note: Mocking syntax has not been carefully considered, but one proof of concept is to use a standard lambda function syntax, as shown below.

```
test mainTest 'main.bicep' = {
params: {
location: 'eastus'
env: 'prod'
}
mocks: {
resourceGroup: () => {location: 'westus'}
}
}
```

### Running Tests, Output, and Results:
Tests can be run through CLI, CI/CD pipelines, and the VS Code integrated debugger. The output result format is standardized regardless of test results so it can be easily and maintainably parsed and interpretted. See output requirements in Milestone 1.

To run tests in the CLI, the user runts the command `bicep test `.

## Proposed Milestones & User Workflows

### Milestones:

1. Introduce a new file type .biceptest in which users author test blocks and corresponding unit-test assert statements that can reference test blocks. Implement (1) client-side test blocks that take params as either objects or as a reference to a .bicepparam file and (2) unit-test assert statement that can reference Bicep template resources within test blocks. **Note: Goal functionality for initial release.**
2. Expand on unit-test functionality to add mocking (see definition) of run-time functionality on the client-side. **Note: Some initial progress made on a proof of concept of mocking.**
3. Expand on client-side functionality to execute static and run-time assert statements as part of the deployment process.
4. Implement End-to-End automatic author à unit test à deploy à post-deployment test à teardown loop for users to be fully confident in the expected and actual results of their deployment.

Throughout the development of each milestone, attention should be paid to support testing in both the **CLI (Command Line Interface) and in CI/CD (Continuous Integration/Continuous Deployment) pipelines.**

### Milestone 1 User Workflow:

1. Users author `.bicep` template file
2. Users author `.biceptest` file* and write `test` blocks to reference `.bicep` files, passing in parameters either as a object or as a *reference to an existing `.bicepparam` file.

```
test mainTest 'main.bicep' = {
params: {
location: 'eastus'
env: 'prod'
}
}
```

3. Users author `assert` statements within the `.biceptest` file referencing resources within `test` blocks.  

```
assert mainServiceAppName = testMain.resources.appServiceApp.name == 'prod-solution-app'
```

4. Users run a client-side test command in `bicep test ` to run assert statements (without deploying code).
5. Test results are outputted. The output summary includes:
- Full breakdown of assertion results for each `test` block in the following format: `test` name followed by **Number of Passed assertions** (ie. `assert` statements) / **Total assertions** for each `test` block
- If 1 or more assertion in the test block Failed or Skipped, output detailed account of `assert` name for all Passed and Failed assertions within a test block
- For all Failed assertions, output Expected vs. Actual values
- Summary of all `test` blocks showing **Total** number of test blocks, number of **Passed** test blocks / number of **Total** tests, number of **Failed** test blocks, and number of **Skipped** test blocks. These results are defined as:
- **Total** tests: All `assert` statements related to the test block including Passed, Failed, and Skipped
- **Passed** test: All `assert` statements related to the test block Passed. Represented by [ ✓ ]
- **Failed** test: At least 1 `assert` statement related to the test block Failed. Represented by [ ✗ ]
- **Skipped** test: At least 1 `assert` statement related to the test block Skipped. Represented by [ - ]
- If any `test` is **Skipped**, display a note explaining the reason for the test being Skipped, such as:
- Note: 1 or more Assertions were skipped because you reference properties only available at run-time.

```
[V] testStorage: Passed 5/5 Assertions
[x] testMain: Passed 1/3 Assertions
[V] Assert 'envInName': Passed
[x] Assert 'latestVersionInName': Failed
Expected: testMain2.resources.appServiceApp.name contains 'v3'
Actual: testMain2.resources.appServiceApp.name == 'v2.09-prod-solution-app'
[-] Assert 'resourceGroupLocation': Skipped

Test Summary: 2 Total - Passed 1/2 Tests
Note: 1 or more Assertions were skipped because you reference properties only available at run-time.
```

Note: `"[ V ]"` represents [ ✓ ]

6. User debugs `.bicep` file corresponding to the Failed assert statements
7. User repeats steps 4-7 until all output Passed

## Open Questions
- Mocking syntax

- Assert statement overlap with policy statements

- Relation to what-if

- Re-occurring AKS connection pain point --> is the real problem how long these deployments take in the first place?

- How to define integration testing in terms of cloud infrastructure?

- How to define E2E testing in terms of cloud infrastructure?

- Will some Bicep users only ever be fully confident with complete deployments?

- Should we have some published, suggested, or automatic tests that test Azure resource constraints?
- From Ariel Silverstein: *Here is an idea to add value to the testing framework. As you may already know, Azure resource property constraints are not enforceable and in most cases are only known by reading documentation or trial and error (deploying something, seeing it fail, discovering that the value in the property is not matching the constraints). If we were able to publish known assertions for these Azure resources and properties and run these assertions as a built-in capability of `bicep test` then we are already adding an intrinsic value to this feature.*
 

Start with the existing experimental testing functionality in issue #11967, then review this proposal's milestones for the intended .biceptest format, test and assert blocks, and the `bicep test ` CLI entry point. The proposal does not name implementation files or tests, and completion is not defined beyond designing and implementing the broader testing framework.

Written by the indexing model from the issue text.