aws / aws/aws-cdk

integ-tests-alpha: cdk tests don't run if we update assertion

Open
#25,191 3 comments 1 reaction 0 assignees View on GitHub
@aws-cdk/assertions bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

```text
"@aws-cdk/integ-runner": "^2.73.0-alpha.0",
"@aws-cdk/integ-tests-alpha": "^2.73.0-alpha.0",
```

Command: `integ-runner --directory ./integ-tests --parallel-regions us-east-1 --update-on-failed`

We created integration tests for our CDK application and there was a snapshot taken on a successful test pass. When I went in and changed some parameters of the integ-tests (changed the assertion expected value from a Dynamo GetItem call) it failed to run the test again and incorrectly passed each subsequent run of the command above. I ran npm run build before running the integ-runner.

I had add the `--force` flag to the integ runner command to get things to run. Filing a bug because this is unexpected behavior and allows for test cases that are incorrect to be passed.

```javascript
integ.assertions
.awsApiCall('DynamoDB', 'getItem', {
TableName: stackUnderTest.tableName,
Key: { id: { S: id } },
})
/**
* Expect the enriched message to be returned.
*/
.expect(
ExpectedResult.objectLike({
Item: {
id: {
S: id,
},
message: {
S: message,
},
additionalAttr: {
S: 'enriched', // Changing this line does not make test run again
},
},
}),
)
```

### Expected Behavior

Tests should be rerun if the integration tests definition change.

### Current Behavior

After changing the assertion in the integration tests definition the tests still pass correctly (only compares snapshot).
No changes were made to application definition.

```bash
> integ-runner --directory ./integ-tests --parallel-regions us-east-1

Verifying integration test snapshots...

UNCHANGED integ.sns-sqs-ddb 5.384s

Snapshot Results:

Tests: 1 passed, 1 total
```

### Reproduction Steps

1. Clone repo: https://github.com/aws-samples/cdk-integ-tests-sample
2. Run `npm i && npm build && npm run integ-test` - should run fine.
3. Update integration test assertion and run `npm run build && npm run integ-test` - passes successfully even though assertion changed.

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.75.1 (build aa30a30)

### Framework Version

_No response_

### Node.js Version

8.6.0

### OS

Mac

### Language

Typescript

### Language Version

Typescript (4.6.3)

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue using the cdk-integ-tests-sample repository and the integ-runner command from the report. First run the integration test, change the additionalAttr assertion, rebuild, and run it again; inspect snapshot verification and the --force behavior. Done means changed integration-test definitions trigger execution rather than being reported unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js, typescript
Domain
cloud, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.