anchore / anchore/syft

Configurable package merge behavior

Open
#3,485 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
9.6k
Forks
962
Avg merge
23h 27m
Merged PRs (30d)
48

Description

**What would you like to be added**:
The user should be able to configure with more nuance:
- how to detect when two packages should be merged
- how to persist package details for merged packages

For instance, today you can assume that any `pkg.Package.Metadata` will be a single struct that represents details for a single package. However, when merging packages today these structs MUST be the same. What is being proposed is to not necessarily require this, so the user can merge similar packages that might have different data cataloged.

This requires us to "join" metadatas like so:

```golang
# assign this to pkg.Package.Metadata
# note: we logically want a map, but not actually so
type JoinedMetadata map[file.CoordinateSet]any
```

This way when merging packages with fuzzier logic we can still keep all information (try not to drop anything).

From a user configuration perspective, this could look like so:

```
package:
merge:
# hash location paths when making package IDs
use-paths: true

# hash location layer information when making package IDs
use-layers: false

# hash package metadata struct when making package IDs
use-metadata: true

use-license: true
use-purl: true
# use....
```

... this is probably a bad example; I think a good part of this issue discussion should be about how the user would specify this.

Another way to do this is to make a single flag instead of a config for this:
```
syft --merge-pkg ENUMVALUE
```
(handwaving at the enum values now)

It's very possible that we should have specific heuristics in specific cataloging cases, so we don't expose these low level options directly, but instead allow or deny a list of heuristics.

Another variant, specify a list of package types to merge:
```
# be more aggressive on deduplication logic for packages of the given types
# note: this does not merge across types, so you can still only merge python packages with other python packages.
syft --merge-pkg-types python,ruby,binary,rpm
```

**Why is this needed**:
Today syft makes a few assumptions about what makes a distinctive package:
- packages with the exact same core + metadata information in different project trees (different paths) are considered to be distinct
- packages with the same core + metadata information in the same path but different layers are considered to NOT be distinct

This allows us to produce SBOMs with the "maximum resolution" so to speak -- you can answer questions about separate project trees (since they are not merged). However, this also has the downside of producing potentially large SBOMs and package graphs when logically they may be for the same application. Merging dependency trees may be ideal for a users use case. We have a few detailed examples of these:
- https://github.com/anchore/syft/issues/1162
- https://github.com/anchore/syft/issues/2559
- https://github.com/anchore/syft/issues/3131
- https://github.com/anchore/syft/issues/3403
- https://github.com/anchore/syft/issues/3535
- (there are probably more related issues!)

This hints that we should make this behavior configurable.

**Additional context**:

Today we have deduplication of OS and binary packages, which is one of the only cases of cross-package-type deduplication behavior (by dropping). How would this be affected by the proposed configuration?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the related issues 1162, 2559, 3131, 3403, and 3535, then identify the package merge and deduplication paths; done means an agreed configuration model that preserves merged metadata and defines its effect on package types.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.