aws / aws/aws-cdk

integ-runner: exit code 1 despite tests passing

Open
#33,279 1 comment 1 reaction 0 assignees View on GitHub
@aws-cdk/integ-runner bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

If a change contains destructive changes, `integ-runner` will exit with exit code `1` with the following message:
```
Tests: 2 passed, 2 total
Error: Some changes were destructive!
at main (~/.nvm/versions/node/v18.17.1/lib/node_modules/@aws-cdk/integ-runner/lib/index.js:10271:11)
```

This happens even when the tests pass and the `--disable-update-workflow` flag is used.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

_No response_

### Expected Behavior

I expect that if the tests pass with `--disable-update-workflow`, that the job succeeds.

### Current Behavior

The job fails with exit code `1`.

### Reproduction Steps

1. Make a destructive change.
2. Run integration tests using `integ-runner --update-on-failed --disable-update-workflow`
3. Check the exit code via `echo $?`

### Possible Solution

Two ideas:
- exit with code `0` to indicate that the tests succeeded without error and that the snapshots have been updated.
- exit with some other code (not `1`) to differentiate this error from an actual test failure.

### Additional Information/Context

In my project I use a CI job which executes the integration tests and commits the new snapshot files. With this bug, I need to run integration tests using the following pattern to suppress this error in order for the new snapshots to be committed:
```
integ-runner ... || true
```

However, given that exit code `1` is used for genuine test failures, those errors are suppressed as well.

### CDK CLI Version

2.174.0

### Framework Version

_No response_

### Node.js Version

v18.17.1

### OS

MacOS

### Language

Python

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the integ-runner command and reproduce the exit status using --update-on-failed --disable-update-workflow, then compare it with a genuine test failure. Review how exit statuses are handled for destructive changes and passing tests; done means successful snapshot updates no longer require || true while real test failures remain detectable.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
cli, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.