aws / aws/aws-cdk

integ-tests: assertion chaining do not fail as expected

Open
#32,604 1 comment 2 reactions 0 assignees View on GitHub
@aws-cdk/assertions bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

The various implementations of `IApiCall` (`awsApiCall`, `invokeFunction` and `httpApiCall`) are used to run assertions on integration test after their deployment. These API or HTTP calls can be reused to run multiple assertions over a single result, either by chaining them or storing the result in a variable and reusing it. Unfortunately, it seems that only the last assertion evaluated for an `IApiCall` instance seem to determine whether the assertion will pass or fail the test.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

Each assertions made to a `IApiCall` construct should be evaluated, and the tests should only pass if every one is valid

### Current Behavior

When running multiple assertions on the same `IApiCall`, only the last assertion determines whether it fails or passes.

### Reproduction Steps

Given the following integration test, based off of [integ.assertions.ts](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/integ-tests-alpha/test/assertions/providers/integ.assertions.ts):

```ts
import { App, CfnResource, Stack } from 'aws-cdk-lib';
import { ExpectedResult, IntegTest } from '../../../lib';

const app = new App();
const stack = new Stack(app, 'Assertions');

const ssmParameter = new CfnResource(stack, 'Utf8Parameter', {
type: 'AWS::SSM::Parameter',
properties: {
Type: 'String',
Value: 'ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ!"#¤%&/()=?`´^*+~_-.,:;<>|',
},
});

const integ = new IntegTest(app, 'AssertionsTest', {
testCases: [stack],
});
```

The following assertions fail, as expected:
```ts
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }));

integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'));

integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }))
.expect(ExpectedResult.objectLike({ Parameter: { Type: 'String' } }))
.expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }));

integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'))
.assertAtPath('Parameter.Type', ExpectedResult.stringLikeRegexp('String'))
.assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'));
```

But the following assertions pass, as their last `expect`/`assertAtPath` calls pass:
```ts
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }))
.expect(ExpectedResult.objectLike({ Parameter: { Type: 'String' } }));

integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'))
.assertAtPath('Parameter.Type', ExpectedResult.stringLikeRegexp('String'));
```

We can see the same issue occurring with `httpApiCall` and `invokeFunction`, as these assertions pass erroneously:
```ts
integ.assertions.httpApiCall(
'https://httpbin.org/status/403',
).expect(ExpectedResult.objectLike({ invalid: 'object' }))
.expect(ExpectedResult.objectLike({ status: 403 }));
```

```ts
const targetFunc = new lambda.Function(stack, 'TargetFunc', {
code: lambda.Code.fromInline('exports.handler = async (event, context) => { return { foo: "bar" }; };'),
handler: 'index.handler',
runtime: lambda.Runtime.NODEJS_LATEST,
});

integ.assertions.invokeFunction({
functionName: targetFunc.functionName,
invocationType: InvocationType.EVENT,
payload: JSON.stringify({ days: 1 }),
}).expect(ExpectedResult.objectLike({ invalid: 'value' }))
.expect(ExpectedResult.objectLike({ StatusCode: 202 }))
.waitForAssertions({
interval: Duration.seconds(30),
totalTimeout: Duration.minutes(90),
});
```

### Possible Solution

The brute and safe solution would be to render these assertion functions effective singletons, by throwing an error if the instance of the `IApiCall` construct is used more than once. This would allow us to at least root out all potential silently failing tests.

### Additional Information/Context

We have multiple instances of multiple assertions being made to a single `IApiCall`, such as:

https://github.com/aws/aws-cdk/blob/034679a6a4f800f8c3db155f6b724eec3ac07d11/packages/@aws-cdk-testing/framework-integ/test/aws-iam/test/integ.imported-role.ts#L65-L80

This might be a regression, but I cannot confirm that this was ever working properly

### CDK CLI Version

v2.173.2 (034679a)

### Framework Version

_No response_

### Node.js Version

v20.11.1

### OS

macOS

### Language

TypeScript

### Language Version

TypeScript ~5.5.2 (internal version)

### Other information

Originally reported by @brandondahler in https://github.com/aws/aws-cdk/pull/32575#discussion_r1893207940

Contributor guide

Open the contributing guide

Research direction

Start with packages/@aws-cdk/integ-tests-alpha/test/assertions/providers/integ.assertions.ts and the IApiCall entry points awsApiCall, httpApiCall, and invokeFunction. Reproduce the chained expect and assertAtPath examples from the issue, then verify that every assertion on one call affects the final test result, including waitForAssertions.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.