aws / aws/aws-cdk

iinteg-tests-alpha: can't have assertions make a deploy fail

Open
#30,513 2 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/assertions effort/medium feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

I want to have Integration tests run both as standalone tests using integ-runner and as part of the regular deployment. Right now, I'm able to do this, but can't get the deployments to fail when tests fail.

### Expected Behavior

Failed integration tests that are part of a regular deployment should make the deployment fail.

### Current Behavior

Deployment shows the failed test output, but succeeds anyways.

### Reproduction Steps

Given the following stack:

```typescript
#import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import createAssertions from "./assertions";

export interface StackProps extends cdk.StackProps {
runAssertions?: boolean
envName?: string
}

export class IdemoStack extends cdk.Stack {
readonly sampleFunction: lambda.Function;
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);

const {runAssertions = true, envName=""} = props??{}
console.log("props: ", props)

this.sampleFunction = new lambda.Function(this, `sampleFunction${envName? "-" + envName:""}`, {
runtime: lambda.Runtime.NODEJS_LATEST,
handler: "index.handler",
code: lambda.Code.fromInline(`
exports.handler = async (event) => {
return true;
};
`),
});
this.sampleFunction.currentVersion

if (runAssertions) {
createAssertions(this)
}
}
}
```
the following integration test file
```typescript
import { App } from "aws-cdk-lib";
import { IdemoStack } from "../lib/idemo-stack";
import { IntegTest } from "@aws-cdk/integ-tests-alpha";
import createAssertions from "../lib/assertions";
import { DeployAssert } from '@aws-cdk/integ-tests-alpha/lib/assertions/private/deploy-assert';

const app = new App();

const stack = new IdemoStack(app, "StackUnderTest", {runAssertions:false, envName:"integration"});

const integTest = new IntegTest(app, "IntegrationTests", {
testCases: [stack],
diffAssets: true,
stackUpdateWorkflow: false,
cdkCommandOptions: {
destroy: {
args: {
force: true,
},
},
},
});

const deployAssert = integTest.assertions;
if (!DeployAssert.isDeployAssert(deployAssert)) {
throw new Error('Expected DeployAssert');
}
const assertionScope = deployAssert.scope;

createAssertions(stack, assertionScope);

app.synth()
```
and the assertions defined like this:
```typescript
import * as integ_tests from "@aws-cdk/integ-tests-alpha";
import { IdemoStack } from "./idemo-stack";
import { CustomResource, Stack } from "aws-cdk-lib";

export default function createAssertions(
stack: IdemoStack,
testStack?: Stack,
) {
const testStack_ = testStack ?? stack;
const val = new integ_tests.LambdaInvokeFunction(
testStack_,
"testSampleFunction",
{
functionName: stack.sampleFunction.functionName,
},
);
val.expect(integ_tests.ExpectedResult.objectLike({ Payload: 'false' }));
}
```

If I run the tests with integ-runner, they fail as expected. But if I do a cdk deploy, the test fails but the deployment succeeds

### Possible Solution

I find a workaround to make the deployment fail as I expect it to. if I add the following to the end of the createAssertions function

```typescript
new integ_tests.EqualsAssertion(testStack_, "assertAssertionPassed", {
failDeployment: true,
expected: integ_tests.ExpectedResult.objectLike({status: "success"}),
actual: integ_tests.ActualResult.fromCustomResource(val.node.findChild("Default") as CustomResource, "assertion")
}).node.tryRemoveChild("AssertionResults")
```

Then the deployment fails when the tests fail

### Additional Information/Context

Ideally, I think there should be a way to add the `failDeployment: true` to the regular expect instead of having to check for the assertion value in a separate assertion

### CDK CLI Version

2.143.1

### Framework Version

_No response_

### Node.js Version

20.12.2

### OS

MacOs

### Language

TypeScript

### Language Version

5.4.5

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the @aws-cdk/integ-tests-alpha assertion entry points shown in the reproduction, especially IntegTest, LambdaInvokeFunction, DeployAssert, and ExpectedResult, then compare integ-runner behavior with cdk deploy. The workaround in createAssertions shows the expected failure path: a failed assertion should cause the regular deployment to fail without requiring a separate EqualsAssertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
devops, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.