DependencyTrack / DependencyTrack/dependency-track

OSV sync activity terminates early when many ecosystems are enabled

Open
#6,971 3 comments 0 reactions 0 assignees View on GitHub
defect integration/osv p2 size/M
Dominant language
Java
Stars
4.2k
Forks
811
Avg merge
8h 39m
Merged PRs (30d)
237

Description

### Current Behavior

When using OSV sync, the actual runtime is dependent on "random" factors such as upstream change activity. Combined with the default activity timeouts (as seen in #6862) this leads to inconsistent behavior.

Generally, there seem to be certain ecosystems that contain more / more complex vulnerabilities and thus take longer to parse in their entirety. Prime examples of this in my observations so far have been:
- `Ubuntu`
- `Debian`
- `Linux`

Specifically the `Ubuntu` ecosystem runtime has been mentioned as causing issues before (#5934). This continues to be a problem in our deployment.

The issue at hand is probably best described as follows:
**All ecosystems are currently processed in a single activity.** This means the more ecosystems are used, the less time each of them has to complete its sync.
What I now observed running 5.0.4 is that the default timeout of 1 hour is not enough for all ecosystems configured on my system to do a full sync.
Depending on the order of execution configured this may mean that some ecosystems are never fetched and thus silently ignored.

For ease of explanation I'll provide the following simplified case with slightly exaggerated durations:
1. Assume a full sync taking a specific amount of time:
- `Ubuntu`: 25m
- `Debian`: 240m
- `Linux`: 15m
2. A partial sync meanwhile is expected to complete faster:
- `Ubuntu`: 2m
- `Debian`: 30m
- `Linux`: 1m
3. The user configures OSV in order `Linux` -> `Debian` -> `Ubuntu`

The above example will obviously fail already in the initial sync. At best `Linux` is completed and persisted, at worst, nothing is persisted if the first ecosystem is the offender. Both `Debian` and `Ubuntu` are never processed. Repeated runs do not fix the issue over time as partial syncs within an ecosystem seem to be discarded from what I can tell.

Should the sum of execution times for a specific run approach the timeout, results become increasingly inconsistent. If upstream changed a lot, a full sync might be triggered, causing an otherwise working pipeline to suddenly fail after part of its execution.

Now obviously, this could be addressed by just increasing the timeout value for the whole activity without caring about total runtime. However, that would somewhat defeat the purpose of having a sensible default timeout at all in my opinion.

I wonder what the solution for this should be:
- Increase the timeout until it works and hope upstream changes don't increase runtime?
- Try to decrease import times by optimizing such that this is no longer an issue?
- Split the activity to perform its work in batches or one-activity-per-ecosystem tasks?

### Steps to Reproduce

1. Configure OSV sync with many sources (big contributors seem to be Linux, Debian and Ubuntu)
2. Start OSV sync
3. Observe sync is aborted after 1 hour (independent of how many "subtasks" actually completed)

### Expected Behavior

OSV sync yields consistent results independent from the number and order of upstream sources.

### Dependency-Track Version

5.x

### Browser

N/A

### Checklist

- [x] I have read and understand the [contributing guidelines](https://github.com/DependencyTrack/dependency-track/blob/main/CONTRIBUTING.md#filing-issues)
- [x] I have checked the [existing issues](https://github.com/DependencyTrack/dependency-track/issues) for whether this defect was already reported

Contributor guide

Open the contributing guide

Research direction

Start by tracing the OSV sync activity and its one-hour activity timeout, then inspect how configured ecosystems are scheduled and how partial results are persisted. Compare runs with several sources, including Linux, Debian, and Ubuntu. Done means every configured ecosystem is processed consistently regardless of order or upstream activity, without silently losing completed work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.