Forge App unit tests take too long to run
- Dominant language
- JavaScript
- Stars
- 400
- Forks
- 89
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 149
Description
### Description
Our [Default build and test](https://github.com/flowforge/flowforge/actions/workflows/build.yml) action is routinely taking 20 minutes.
Likewise our postgres tests - although they also suffer from more regularly OOM errors (#1960).
They don't take as long when running locally, but it is still many minutes.
This is only going to get worse as we write more tests (I'm told removing tests isn't the right answer....)
We need to change our approach.
Currently the typical pattern is for each individual test case to be run on a *clean* FlowForge instance. This means each test has to initialise the instance before it can be run. This setup time doesn't get reflected in the timings mocha provides for individual tests.
I've previously looked at ways to make this more efficient, for example reseting the database rather than recreating it. That had very limited success.
The main goal needs to be to minimise how many times the test suites recreate the forge app.
The general principle should be:
1. Each test suite (ie a _spec file) remains responsible for creating a forge app instance to test against.
We must not end up needing to run the _spec files in a specific order, or being unable to run an individual _spec file because it assumes another has run before it.
3. Rather than recreating that app instance for each individual test, the tests should be able to run against the same instance.
For many existing tests that will require them to be updated to use test-specific resource names - so it doesn't matter that a resource already exists from a previous test.
5. The tests must remain independent - even though they share a forge app instance. Minimise cases where TestB will only pass if TestA has run. That will make for brittle tests when adding new test cases in between etc.
Some tests will continue to require a clean app to test against - for example where specific platform-wide configurations are needed (eg billing config). The goal just needs to be to minimise how many times that's done.
---
This is a list of all spec files with a count of the test cases within it. This will help prioritise the files that should be refactored (more tests === more times the app is reinitialised).
- [x] 53 ./routes/api/project_spec.js
- [x] 48 ./comms/authRoutes_spec.js
- [x] 31 ./routes/api/device_spec.js
- [ ] 30 ./containers/index_spec.js
- [x] 29 ./auditLog/team_spec.js
- [x] 27 ./ee/routes/billing/index_spec.js
- [x] 25 ./routes/api/users_spec.js
- [x] 25 ./routes/api/user_spec.js
- [ ] 23 ./ee/lib/billing/index_spec.js
- [x] 23 ./db/controllers/User_spec.js
- [x] 22 ./routes/api/teamMembers_spec.js
- [x] 22 ./routes/api/stack_spec.js
- [x] 21 ./routes/api/application_spec.js
- [x] 18 ./db/models/User_spec.js
- [x] 18 ./auditLog/user_spec.js
- [x] 17 ./routes/storage/index_spec.js
- [x] 17 ./auditLog/project_spec.js
- [x] 17 ./auditLog/formatters_spec.js
- [ ] 16 ./routes/api/team_spec.js
- [x] 14 ./licensing/index_spec.js
- [ ] 14 ./ee/db/controllers/Subscription_spec.js
- [x] 13 ./routes/auth/index_spec.js
- [ ] 13 ./routes/api/teamDevices_spec.js
- [x] 13 ./ee/routes/sharedLibrary/index_spec.js
- [x] 13 ./ee/lib/sso/index_spec.js
- [x] 12 ./db/controllers/AccessToken_spec.js
- [ ] 11 ./routes/api/settings_spec.js
- [ ] 11 ./routes/api/projectType_spec.js
- [x] 11 ./db/controllers/Team_spec.js
- [x] 11 ./db/controllers/Project_spec.js
- [x] 11 ./db/controllers/ProjectTemplate_spec.js
- [x] 11 ./auditLog/platform_spec.js
- [x] 10 ./routes/api/projectSnapshots_spec.js
- [x] 9 ./ee/routes/sso/index_spec.js
- [x] 7 ./routes/api/teamInvitations_spec.js
- [x] 7 ./lib/validate_spec.js
- [x] 6 ./licensing/loader_spec.js
- [x] 6 ./db/controllers/Invitation_spec.js
- [ ] 5 ./ee/lib/billing/trialTask_spec.js
- [x] 5 ./db/models/Project_spec.js
- [x] 5 ./db/models/Device_spec.js
- [x] 3 ./routes/logging/index_spec.js
- [x] 3 ./routes/api/projectDevices_spec.js
- [x] 3 ./db/models/Team_spec.js
- [x] 3 ./db/controllers/ProjectSnapshot_spec.js
- [x] 3 ./db/controllers/Device_spec.js
- [ ] 2 ./ee/routes/api/project_spec.js
- [x] 2 ./db/controllers/StorageSettings_spec.js
Contributor guide
Assessment
This issue has not been assessed yet.