DependencyTrack / DependencyTrack/dependency-track

Performance degradation due to possibly internal workers not being able to deal with a big amount of MavenMetaAnalyzer timeout errors

Open
#5,636 6 comments 0 reactions 0 assignees View on GitHub
defect in triage
Dominant language
Java
Stars
4.2k
Forks
811
Avg merge
9h 4m
Merged PRs (30d)
229

Description

### Current Behavior

Hello. Sorry if there is a similar issue but i coudn't find any.
So, our dependency-track experienced 2 episodes on the last month with a big performance degradation but there wasn't sufficiente information on the logs. The first happened on november 18 (which coincided with that big unavailability on cloud flare) e the other on December 9. In both of the situations the problem was solved by restarting the api-server container but this isn't an effective strategy because we have a high demand enviroment with security guardrails being controlled by Dependency-track results

We have custom scripts to run scans in the development pipelines and it was reported to us that this scans were taking far too long and failing. The frontend was fine and the APIs were responding fine as well... The CPU and memory usage were also very low (our vm has an Intel Xeon with 10 cores available to the VM and 32 GB of RAM).

When I looked at the logs, I coudn't find anything out of the ordinary, but I pinpointed the most likely moment where the degration started:

Image

As you can see by the print, it started running the RepositoryMetaAnalyzerTask for the entire portfolio and this was followed only with this MavenMetaAnalyzer error.

I even tried triggering as metrics update for a specific project as a test and it coudn't even start the task:

2025-12-09 15:15:34,303 WARN [PortfolioMetricsUpdateTask] Updating metrics for projects 3603-3591 took longer than expected (15m); Proceeding with potentially
stale data

2025-12-09 15:13:34,205 ERROR [MavenMetaAnalyzer] Request failure
java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source)
at java.base/java.net.Socket$SocketInputStream.read(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:111)
at org.dependencytrack.tasks.repositories.MavenMetaAnalyzer.analyze(MavenMetaAnalyzer.java:81)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:193)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:137)
at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:142)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:113)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

This MavenMetaAnalyzer errors happens sometimes even with when Dependency-Track is running fine and I think it could be a combination of the status of Maven endpoint + the fact that our Dependency-Track is on our private network so it is expected some latency on that. But the issue that I am reporting is not related to the Maven error specifically, but on how Dependency-Track deal with a possbility of a lot of timeout when running the RepositoryMetaAnalyzerTask.

The only errors showing on the log were this Maven timeout errors and everything else was fine, CPU, memory and the APIs were responding normally, so it was a performance degradation specifically on the internal tasks, probably due to a lot of timeout tasks that impacted other tasks.

Some other thing to note that at this moment (as you can see on the print) Dependency-Track was also running the periodic metrics update task and that probably also contributed for that perfomance degradation. I already scheduled the metrics task to run on a low activity time for us so to avoid this problem.

On researching this problem, I came to the conclusion that the most plausible explanation is that the internal workers were stuck trying to communicate to maven and newer tasks coudn't be assigned to this workers. I also noticed that we were using the default HTTP timeout settings:

# Optional HTTP Outbound Connection Timeout Settings. All values are in seconds.
- ALPINE_HTTP_TIMEOUT_CONNECTION=15
- ALPINE_HTTP_TIMEOUT_SOCKET=15
# - ALPINE_HTTP_TIMEOUT_POOL=60

So we reduced this time to 15s and then 25s for testing. As I said, out application is on the internal network so it has to go through the proxy, firewalls and a lot more, so i think if a go lower than 15s it could be a problem. But as I said, the problem was not the maven error but the performance degradation due to probably repeated errors.

Unfortunately I don't have a lot more evidence as there was anything specific on logs.

This enviroment is pretty stable and have been running since 2023, with no major problems aside from eventual bugs that were fixed by updating the version. It is also a pretty big enviroment with arroung 4100 projects, 3000 weekly bom uploads and 1699399 components. This specific problem never happened before and usually it is pretty fast, but we have been expanding our projects gradually on 2025 so this increase in workload probably also contributed in a way to this situation... Like I said, we monitor CPU and Memory usage and it is low most of the time.

### Steps to Reproduce

1. Make HTTP tasks to fail repeatly due to network issues during the RepositoryMetaAnalyzer task
2. Observe the behavior of other tasks and workers

### Expected Behavior

To dependency-track being able to deal with the amount of HTTP timeout errors without ocuppying all available workers and leaving other newer tasks to hang

### Dependency-Track Version

4.12.x

### Dependency-Track Distribution

Container Image

### Database Server

PostgreSQL

### Database Server Version

psql (PostgreSQL) 14.3 (Debian 14.3-1.pgdg110+1)

### Browser

Google Chrome

### Checklist

- [x] I have read and understand the [contributing guidelines](https://github.com/DependencyTrack/dependency-track/blob/master/CONTRIBUTING.md#filing-issues)
- [x] I have checked the [existing issues](https://github.com/DependencyTrack/dependency-track/issues) for whether this defect was already reported

Contributor guide

Open the contributing guide

Research direction

Start by tracing the stack entries in AbstractMetaAnalyzer.java:111, MavenMetaAnalyzer.java:81, and RepositoryMetaAnalyzerTask.java:113-193, then reproduce repeated HTTP timeouts during repository analysis. Done means other newer tasks continue to run instead of hanging when RepositoryMetaAnalyzerTask encounters many timeout errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.