DependencyTrack / DependencyTrack/dependency-track
Improve CVE parsing performance if database latency is present
- Dominant language
- Java
- Stars
- 4.2k
- Forks
- 811
- Avg merge
- 8h 39m
- Merged PRs (30d)
- 237
Description
### Current Behavior
Dependency track does 4000 commits per seconds for an hour to parse the CVE database when running in AWS EC2 m5d.xlarge with a local PostgreSQL on SSD storage. When switching to RDS PostgreSQL for the reason of having proper database management, backups, etc. the performance drops significantly, the commit rate is around 1500 commits / s.
In general, an application loading data into an SQL server should not commit the transaction so frequently. Even for a single record multiple commits are issued in DataNucleus, one for each field update.
### Proposed Behavior
Batch inserts and updates, and do commit less frequently to improve performance over database links with more latency than a local database.
### Checklist
- [X] I have read and understand the [contributing guidelines](https://github.com/DependencyTrack/dependency-track/blob/master/CONTRIBUTING.md#filing-issues)
- [X] I have checked the [existing issues](https://github.com/DependencyTrack/dependency-track/issues) for whether this enhancement was already requested
Contributor guide
Assessment
This issue has not been assessed yet.