DependencyTrack / DependencyTrack/dependency-track
OSV sync activity terminates early when many ecosystems are enabled
- Dominant language
- Java
- Stars
- 4.2k
- Forks
- 811
- Avg merge
- 8h 39m
- Merged PRs (30d)
- 237
Description
### Current Behavior
When using OSV sync, the actual runtime is dependent on "random" factors such as upstream change activity. Combined with the default activity timeouts (as seen in #6862) this leads to inconsistent behavior.
Generally, there seem to be certain ecosystems that contain more / more complex vulnerabilities and thus take longer to parse in their entirety. Prime examples of this in my observations so far have been:
- `Ubuntu`
- `Debian`
- `Linux`
Specifically the `Ubuntu` ecosystem runtime has been mentioned as causing issues before (#5934). This continues to be a problem in our deployment.
The issue at hand is probably best described as follows:
**All ecosystems are currently processed in a single activity.** This means the more ecosystems are used, the less time each of them has to complete its sync.
What I now observed running 5.0.4 is that the default timeout of 1 hour is not enough for all ecosystems configured on my system to do a full sync.
Depending on the order of execution configured this may mean that some ecosystems are never fetched and thus silently ignored.
For ease of explanation I'll provide the following simplified case with slightly exaggerated durations:
1. Assume a full sync taking a specific amount of time:
- `Ubuntu`: 25m
- `Debian`: 240m
- `Linux`: 15m
2. A partial sync meanwhile is expected to complete faster:
- `Ubuntu`: 2m
- `Debian`: 30m
- `Linux`: 1m
3. The user configures OSV in order `Linux` -> `Debian` -> `Ubuntu`
The above example will obviously fail already in the initial sync. At best `Linux` is completed and persisted, at worst, nothing is persisted if the first ecosystem is the offender. Both `Debian` and `Ubuntu` are never processed. Repeated runs do not fix the issue over time as partial syncs within an ecosystem seem to be discarded from what I can tell.
Should the sum of execution times for a specific run approach the timeout, results become increasingly inconsistent. If upstream changed a lot, a full sync might be triggered, causing an otherwise working pipeline to suddenly fail after part of its execution.
Now obviously, this could be addressed by just increasing the timeout value for the whole activity without caring about total runtime. However, that would somewhat defeat the purpose of having a sensible default timeout at all in my opinion.
I wonder what the solution for this should be:
- Increase the timeout until it works and hope upstream changes don't increase runtime?
- Try to decrease import times by optimizing such that this is no longer an issue?
- Split the activity to perform its work in batches or one-activity-per-ecosystem tasks?
### Steps to Reproduce
1. Configure OSV sync with many sources (big contributors seem to be Linux, Debian and Ubuntu)
2. Start OSV sync
3. Observe sync is aborted after 1 hour (independent of how many "subtasks" actually completed)
### Expected Behavior
OSV sync yields consistent results independent from the number and order of upstream sources.
### Dependency-Track Version
5.x
### Browser
N/A
### Checklist
- [x] I have read and understand the [contributing guidelines](https://github.com/DependencyTrack/dependency-track/blob/main/CONTRIBUTING.md#filing-issues)
- [x] I have checked the [existing issues](https://github.com/DependencyTrack/dependency-track/issues) for whether this defect was already reported
Contributor guide
Research direction
Start by tracing the OSV sync activity and its one-hour activity timeout, then inspect how configured ecosystems are scheduled and how partial results are persisted. Compare runs with several sources, including Linux, Debian, and Ubuntu. Done means every configured ecosystem is processed consistently regardless of order or upstream activity, without silently losing completed work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100