DependencyTrack / DependencyTrack/dependency-track

Low performance with analyzers

Open
#6,057 3 comments 0 reactions 0 assignees View on GitHub
defect integration/trivy performance size/S
Dominant language
Java
Stars
4.2k
Forks
811
Avg merge
9h 4m
Merged PRs (30d)
229

Description

### Current Behavior

Analyzers take a long time to complete, ~10 seconds per project or more on average.

### Steps to Reproduce

Scan a project with a few hundred components.

### Expected Behavior

Analyzers should be faster.

### Analysis suggestion

From watching the live queries of the DB, 90% or more of the queries are basically selecting only components. It seems to include a lot of joins, and from the initial analysis, it suggests that the core issue is single queries for single components during analysis tasks - my current suspicion is that queries like https://github.com/DependencyTrack/dependency-track/blob/00ea12da532f215cf729aca04f81d6c156f3105d/src/main/java/org/dependencytrack/tasks/scanners/TrivyAnalysisTask.java#L487 are causing this and should be replaced with batch queries. Every query I saw in the live queries is not taking much of time (<100ms), but when you have 2k projects with roughly 200 components each, and assume an average runtime of 100ms per query, plus analyzer overhead of around 100ms, this adds up to around 22 hours to scan the whole portfolio. The idea is to replace single queries with batch queries to avoid repeated parsing and query optimization. But this is only an assumption.

Suggestions are welcome, since I don't trust my own analysis yet, and I believe there might be other causes for this.

### Dependency-Track Version

4.13.6

### Dependency-Track Distribution

Container Image

### Database Server

PostgreSQL

### Database Server Version

N/A

### Browser

N/A

### Checklist

- [x] I have read and understand the [contributing guidelines](https://github.com/DependencyTrack/dependency-track/blob/master/CONTRIBUTING.md#filing-issues)
- [x] I have checked the [existing issues](https://github.com/DependencyTrack/dependency-track/issues) for whether this defect was already reported

Contributor guide

Open the contributing guide

Research direction

Start with src/main/java/org/dependencytrack/tasks/scanners/TrivyAnalysisTask.java near line 487 in the linked revision. Reproduce a scan of a project with a few hundred components on PostgreSQL and measure query counts and analyzer runtime before treating batching as the solution. Done means demonstrably faster analysis without changing results; the issue supplies no numerical performance target.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
43/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.