iinteg-tests-alpha: can't have assertions make a deploy fail
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
I want to have Integration tests run both as standalone tests using integ-runner and as part of the regular deployment. Right now, I'm able to do this, but can't get the deployments to fail when tests fail.
### Expected Behavior
Failed integration tests that are part of a regular deployment should make the deployment fail.
### Current Behavior
Deployment shows the failed test output, but succeeds anyways.
### Reproduction Steps
Given the following stack:
```typescript
#import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import createAssertions from "./assertions";
export interface StackProps extends cdk.StackProps {
runAssertions?: boolean
envName?: string
}
export class IdemoStack extends cdk.Stack {
readonly sampleFunction: lambda.Function;
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const {runAssertions = true, envName=""} = props??{}
console.log("props: ", props)
this.sampleFunction = new lambda.Function(this, `sampleFunction${envName? "-" + envName:""}`, {
runtime: lambda.Runtime.NODEJS_LATEST,
handler: "index.handler",
code: lambda.Code.fromInline(`
exports.handler = async (event) => {
return true;
};
`),
});
this.sampleFunction.currentVersion
if (runAssertions) {
createAssertions(this)
}
}
}
```
the following integration test file
```typescript
import { App } from "aws-cdk-lib";
import { IdemoStack } from "../lib/idemo-stack";
import { IntegTest } from "@aws-cdk/integ-tests-alpha";
import createAssertions from "../lib/assertions";
import { DeployAssert } from '@aws-cdk/integ-tests-alpha/lib/assertions/private/deploy-assert';
const app = new App();
const stack = new IdemoStack(app, "StackUnderTest", {runAssertions:false, envName:"integration"});
const integTest = new IntegTest(app, "IntegrationTests", {
testCases: [stack],
diffAssets: true,
stackUpdateWorkflow: false,
cdkCommandOptions: {
destroy: {
args: {
force: true,
},
},
},
});
const deployAssert = integTest.assertions;
if (!DeployAssert.isDeployAssert(deployAssert)) {
throw new Error('Expected DeployAssert');
}
const assertionScope = deployAssert.scope;
createAssertions(stack, assertionScope);
app.synth()
```
and the assertions defined like this:
```typescript
import * as integ_tests from "@aws-cdk/integ-tests-alpha";
import { IdemoStack } from "./idemo-stack";
import { CustomResource, Stack } from "aws-cdk-lib";
export default function createAssertions(
stack: IdemoStack,
testStack?: Stack,
) {
const testStack_ = testStack ?? stack;
const val = new integ_tests.LambdaInvokeFunction(
testStack_,
"testSampleFunction",
{
functionName: stack.sampleFunction.functionName,
},
);
val.expect(integ_tests.ExpectedResult.objectLike({ Payload: 'false' }));
}
```
If I run the tests with integ-runner, they fail as expected. But if I do a cdk deploy, the test fails but the deployment succeeds
### Possible Solution
I find a workaround to make the deployment fail as I expect it to. if I add the following to the end of the createAssertions function
```typescript
new integ_tests.EqualsAssertion(testStack_, "assertAssertionPassed", {
failDeployment: true,
expected: integ_tests.ExpectedResult.objectLike({status: "success"}),
actual: integ_tests.ActualResult.fromCustomResource(val.node.findChild("Default") as CustomResource, "assertion")
}).node.tryRemoveChild("AssertionResults")
```
Then the deployment fails when the tests fail
### Additional Information/Context
Ideally, I think there should be a way to add the `failDeployment: true` to the regular expect instead of having to check for the assertion value in a separate assertion
### CDK CLI Version
2.143.1
### Framework Version
_No response_
### Node.js Version
20.12.2
### OS
MacOs
### Language
TypeScript
### Language Version
5.4.5
### Other information
_No response_
Contributor guide
Research direction
Start with the @aws-cdk/integ-tests-alpha assertion entry points shown in the reproduction, especially IntegTest, LambdaInvokeFunction, DeployAssert, and ExpectedResult, then compare integ-runner behavior with cdk deploy. The workaround in createAssertions shows the expected failure path: a failed assertion should cause the regular deployment to fail without requiring a separate EqualsAssertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- devops, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100