cosmos / cosmos/interchain-security
[Epic]: Refactor e2e tests to increase coverage and reduce complexity
- Dominant language
- Go
- Stars
- 195
- Forks
- 166
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Currently, the e2e tests consists of the following steps (see [tests/e2e/README.md](https://github.com/cosmos/interchain-security/blob/main/tests/e2e/README.md) for more details):
- start a docker container
- run a defined sequence of actions with expected states
- after each action, query the system state and compare against the expected state
Also, there are a set of predefined test cases (more than 20) that can be executed directly. Run `go run ./tests/e2e/... --help` for details.
Although at the high level this e2e framework is good, there are several issues that need to be addressed:
- Some of the tests are non-deterministic as they depend on timing assumptions. As a result, tests are sometimes failing and need to be rerun.
- Most of the existing test cases have unnecessary actions resulting in tests with a high number of steps (e.g., the `happy-path` has 39 steps).
- This makes the testing time long. As a result, the non-deterministic aspect has an even higher impact, e.g., if step 38 in the `happy-path` test case is randomly failing, the entire test case needs to be retried.
- This also makes the tests complex and hard to modify and debug, e.g., there is no reason to start consumer chains or create IBC channels to test the consumer lifecycle on the provider.
- Many of the test cases are not part of the CI or the nightly tests, which means that are rarely run.
## Closing criteria
The tasks below are completed.
## Task list
```[tasklist]
### Must have
- [x] Create simpler actions, i.e., one task performed per action (e.g., start only the provider).
- [ ] Create EPIC with test cases that require both the provider and a consumer (e.g., CCV channel handshake)
- [ ] https://github.com/cosmos/interchain-security/issues/2315
- [ ] Create EPIC with test cases that require only the consumer (there might be none)
- [x] Replace long test cases with shorter ones that address a single concern
- [ ] Enable multiple test cases to run in parallel (one failing test case should not affect others)
- [ ] Remove test cases that are checking protocol logic (the scope of e2e testing is making sure the API works)
```
```[tasklist]
### Questions
- [ ] Is the interchain testing framework better?
- [ ] How can we reduce non-determinism in certain actions?
- [ ] What are the pain points with compatibility tests?
```
```[tasklist]
### Nice to have
- [ ] https://github.com/cosmos/interchain-security/issues/1325
- [ ] https://github.com/cosmos/interchain-security/issues/1487
```
```[tasklist]
### Others (to triage)
- [ ] https://github.com/cosmos/interchain-security/issues/2198
- [ ] https://github.com/cosmos/interchain-security/issues/1568
- [ ] https://github.com/cosmos/interchain-security/issues/1135
- [ ] https://github.com/cosmos/interchain-security/issues/1132
- [ ] https://github.com/cosmos/interchain-security/issues/877
- [ ] https://github.com/cosmos/interchain-security/issues/1296
```
Contributor guide
Assessment
This issue has not been assessed yet.