DependencyTrack / DependencyTrack/dependency-track
Low performance with analyzers
- Dominant language
- Java
- Stars
- 4.2k
- Forks
- 811
- Avg merge
- 9h 4m
- Merged PRs (30d)
- 229
Description
### Current Behavior
Analyzers take a long time to complete, ~10 seconds per project or more on average.
### Steps to Reproduce
Scan a project with a few hundred components.
### Expected Behavior
Analyzers should be faster.
### Analysis suggestion
From watching the live queries of the DB, 90% or more of the queries are basically selecting only components. It seems to include a lot of joins, and from the initial analysis, it suggests that the core issue is single queries for single components during analysis tasks - my current suspicion is that queries like https://github.com/DependencyTrack/dependency-track/blob/00ea12da532f215cf729aca04f81d6c156f3105d/src/main/java/org/dependencytrack/tasks/scanners/TrivyAnalysisTask.java#L487 are causing this and should be replaced with batch queries. Every query I saw in the live queries is not taking much of time (<100ms), but when you have 2k projects with roughly 200 components each, and assume an average runtime of 100ms per query, plus analyzer overhead of around 100ms, this adds up to around 22 hours to scan the whole portfolio. The idea is to replace single queries with batch queries to avoid repeated parsing and query optimization. But this is only an assumption.
Suggestions are welcome, since I don't trust my own analysis yet, and I believe there might be other causes for this.
### Dependency-Track Version
4.13.6
### Dependency-Track Distribution
Container Image
### Database Server
PostgreSQL
### Database Server Version
N/A
### Browser
N/A
### Checklist
- [x] I have read and understand the [contributing guidelines](https://github.com/DependencyTrack/dependency-track/blob/master/CONTRIBUTING.md#filing-issues)
- [x] I have checked the [existing issues](https://github.com/DependencyTrack/dependency-track/issues) for whether this defect was already reported
Contributor guide
Research direction
Start with src/main/java/org/dependencytrack/tasks/scanners/TrivyAnalysisTask.java near line 487 in the linked revision. Reproduce a scan of a project with a few hundred components on PostgreSQL and measure query counts and analyzer runtime before treating batching as the solution. Done means demonstrably faster analysis without changing results; the issue supplies no numerical performance target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 43/100