integ-tests-alpha: cdk tests don't run if we update assertion
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
```text
"@aws-cdk/integ-runner": "^2.73.0-alpha.0",
"@aws-cdk/integ-tests-alpha": "^2.73.0-alpha.0",
```
Command: `integ-runner --directory ./integ-tests --parallel-regions us-east-1 --update-on-failed`
We created integration tests for our CDK application and there was a snapshot taken on a successful test pass. When I went in and changed some parameters of the integ-tests (changed the assertion expected value from a Dynamo GetItem call) it failed to run the test again and incorrectly passed each subsequent run of the command above. I ran npm run build before running the integ-runner.
I had add the `--force` flag to the integ runner command to get things to run. Filing a bug because this is unexpected behavior and allows for test cases that are incorrect to be passed.
```javascript
integ.assertions
.awsApiCall('DynamoDB', 'getItem', {
TableName: stackUnderTest.tableName,
Key: { id: { S: id } },
})
/**
* Expect the enriched message to be returned.
*/
.expect(
ExpectedResult.objectLike({
Item: {
id: {
S: id,
},
message: {
S: message,
},
additionalAttr: {
S: 'enriched', // Changing this line does not make test run again
},
},
}),
)
```
### Expected Behavior
Tests should be rerun if the integration tests definition change.
### Current Behavior
After changing the assertion in the integration tests definition the tests still pass correctly (only compares snapshot).
No changes were made to application definition.
```bash
> integ-runner --directory ./integ-tests --parallel-regions us-east-1
Verifying integration test snapshots...
UNCHANGED integ.sns-sqs-ddb 5.384s
Snapshot Results:
Tests: 1 passed, 1 total
```
### Reproduction Steps
1. Clone repo: https://github.com/aws-samples/cdk-integ-tests-sample
2. Run `npm i && npm build && npm run integ-test` - should run fine.
3. Update integration test assertion and run `npm run build && npm run integ-test` - passes successfully even though assertion changed.
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
2.75.1 (build aa30a30)
### Framework Version
_No response_
### Node.js Version
8.6.0
### OS
Mac
### Language
Typescript
### Language Version
Typescript (4.6.3)
### Other information
_No response_
Contributor guide
Research direction
Reproduce the issue using the cdk-integ-tests-sample repository and the integ-runner command from the report. First run the integration test, change the additionalAttr assertion, rebuild, and run it again; inspect snapshot verification and the --force behavior. Done means changed integration-test definitions trigger execution rather than being reported unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, node.js, typescript
- Domain
- cloud, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100