loopbackio / loopbackio/loopback-next
Move slow tests from `npm test` to a different test task
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 5.1k
- Forks
- 1.1k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 27
Description
Most of our test suite is running pretty fast, with individual tests finishing under 100ms. However, there are also several slow tests that take more than 1 second to finish. These tests are typically end-to-end/acceptance-level tests that are running running child processes, `npm install` and so on. As a result, `npm test` takes too long to complete (about 3 minutes on my machine).
Let's find a way how to reorganize our code base so that these slow tests can be put into "acceptance" or "end-to-end" group, excluded from `npm test` and have a dedicated CI job to run them in parallel with other CI tasks, as we are already doing for `acceptance/repository-mysql` and friends. (Alternatively, we can run the slow acceptance tests after the faster tests passed succesfully.)
1. For example, we can introduce new package(s) in the `acceptance` folder, e.g. `acceptance/cli-e2e-tests` and move the slow tests there. This should be easy to implement, but requires more work to introduce e2e tests for new packages.
2. Create a new directory-layout convention for acceptance/end-to-end tests, so that we can keep e2e tests inside each package alongside other tests, but still be able to exclude these tests from monorepo-level `npm test`. For example, we can introduce `src/__acceptance__` directory for such tests.
Personally, I think the second option is much better.
## List of very slow tests
Tests that take longer than one second to complete, sorted from the slowest:
- app-generator (SLOW) passes `npm test` for the generated project: 12441ms
- Benchmark (SLOW) works: 4352ms
- lb4 relation generates model relation for existing property name verifies that a preexisting property will be overwritten: 2675ms
- lb4 relation add new controller to existing index file check if the controller exported to index file : 2026ms
- lb4 relation add controller to existing index file only once check if the controller exported to index file only once: 1961ms
- cloneExampleFromGitHub (SLOW) extracts project files: 1869ms
- lb4 relation updates property decorator when property already exist in the model: 1566ms
- lb4 relation HasOne rejects relation when source key already exist in the model: 1362ms
- lb4 relation HasOne Execute relation with existing relation name rejects if the relation name already exists in the repository: 1344ms
- lb4 relation HasMany Execute relation with existing relation name rejects if the relation name already exists in the repository: 1325ms
- lb4 relation rejects relation when destination model doesn't have primary Key: 1320ms
- tsdocs runs api-extractor: 1280ms
- lb4 relation HasMany rejects relation when source key already exist in the model: 1254ms
- lb4 copyright with git updates copyright/license headers with options: 1239ms
- lb4 copyright for monorepo updates copyright/license headers: 1205ms
- build compiles ts files: 1046ms
- Application rejects division by zero with 412 error: 1022ms
## Nice to have
- Investigate why are `lb4 relation` tests so slow. Do they have expensive setup? Can we find a more efficient way how to test the relation generator?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the monorepo-level `npm test` configuration and the existing `acceptance/repository-mysql` CI setup. Review the listed slow tests and compare the `acceptance/cli-e2e-tests` and `src/__acceptance__` layout options. Done means fast tests no longer run in `npm test`, while a dedicated acceptance task or CI job runs the slow tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, typescript
- Domain
- build-system, ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100