integ-tests: assertion chaining do not fail as expected
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
The various implementations of `IApiCall` (`awsApiCall`, `invokeFunction` and `httpApiCall`) are used to run assertions on integration test after their deployment. These API or HTTP calls can be reused to run multiple assertions over a single result, either by chaining them or storing the result in a variable and reusing it. Unfortunately, it seems that only the last assertion evaluated for an `IApiCall` instance seem to determine whether the assertion will pass or fail the test.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Version
_No response_
### Expected Behavior
Each assertions made to a `IApiCall` construct should be evaluated, and the tests should only pass if every one is valid
### Current Behavior
When running multiple assertions on the same `IApiCall`, only the last assertion determines whether it fails or passes.
### Reproduction Steps
Given the following integration test, based off of [integ.assertions.ts](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/integ-tests-alpha/test/assertions/providers/integ.assertions.ts):
```ts
import { App, CfnResource, Stack } from 'aws-cdk-lib';
import { ExpectedResult, IntegTest } from '../../../lib';
const app = new App();
const stack = new Stack(app, 'Assertions');
const ssmParameter = new CfnResource(stack, 'Utf8Parameter', {
type: 'AWS::SSM::Parameter',
properties: {
Type: 'String',
Value: 'ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ!"#¤%&/()=?`´^*+~_-.,:;<>|',
},
});
const integ = new IntegTest(app, 'AssertionsTest', {
testCases: [stack],
});
```
The following assertions fail, as expected:
```ts
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }));
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'));
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }))
.expect(ExpectedResult.objectLike({ Parameter: { Type: 'String' } }))
.expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }));
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'))
.assertAtPath('Parameter.Type', ExpectedResult.stringLikeRegexp('String'))
.assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'));
```
But the following assertions pass, as their last `expect`/`assertAtPath` calls pass:
```ts
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).expect(ExpectedResult.objectLike({ Parameter: { Value: 'Invalid value' } }))
.expect(ExpectedResult.objectLike({ Parameter: { Type: 'String' } }));
integ.assertions.awsApiCall('SSM', 'getParameter', {
Name: ssmParameter.ref,
WithDecryption: true,
}).assertAtPath('Parameter.Value', ExpectedResult.stringLikeRegexp('Invalid value'))
.assertAtPath('Parameter.Type', ExpectedResult.stringLikeRegexp('String'));
```
We can see the same issue occurring with `httpApiCall` and `invokeFunction`, as these assertions pass erroneously:
```ts
integ.assertions.httpApiCall(
'https://httpbin.org/status/403',
).expect(ExpectedResult.objectLike({ invalid: 'object' }))
.expect(ExpectedResult.objectLike({ status: 403 }));
```
```ts
const targetFunc = new lambda.Function(stack, 'TargetFunc', {
code: lambda.Code.fromInline('exports.handler = async (event, context) => { return { foo: "bar" }; };'),
handler: 'index.handler',
runtime: lambda.Runtime.NODEJS_LATEST,
});
integ.assertions.invokeFunction({
functionName: targetFunc.functionName,
invocationType: InvocationType.EVENT,
payload: JSON.stringify({ days: 1 }),
}).expect(ExpectedResult.objectLike({ invalid: 'value' }))
.expect(ExpectedResult.objectLike({ StatusCode: 202 }))
.waitForAssertions({
interval: Duration.seconds(30),
totalTimeout: Duration.minutes(90),
});
```
### Possible Solution
The brute and safe solution would be to render these assertion functions effective singletons, by throwing an error if the instance of the `IApiCall` construct is used more than once. This would allow us to at least root out all potential silently failing tests.
### Additional Information/Context
We have multiple instances of multiple assertions being made to a single `IApiCall`, such as:
https://github.com/aws/aws-cdk/blob/034679a6a4f800f8c3db155f6b724eec3ac07d11/packages/@aws-cdk-testing/framework-integ/test/aws-iam/test/integ.imported-role.ts#L65-L80
This might be a regression, but I cannot confirm that this was ever working properly
### CDK CLI Version
v2.173.2 (034679a)
### Framework Version
_No response_
### Node.js Version
v20.11.1
### OS
macOS
### Language
TypeScript
### Language Version
TypeScript ~5.5.2 (internal version)
### Other information
Originally reported by @brandondahler in https://github.com/aws/aws-cdk/pull/32575#discussion_r1893207940
Contributor guide
Research direction
Start with packages/@aws-cdk/integ-tests-alpha/test/assertions/providers/integ.assertions.ts and the IApiCall entry points awsApiCall, httpApiCall, and invokeFunction. Reproduce the chained expect and assertAtPath examples from the issue, then verify that every assertion on one call affects the final test result, including waitForAssertions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100