l3montree-dev / l3montree-dev/devguard

Implement SBOM Merkle tree for minimal storage requirement of component dependencies table

Open
#2,780 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request
Dominant language
Go
Stars
161
Forks
43
Avg merge
1d 8h
Merged PRs (30d)
37

Description

Proposal: Content-addressed (Merkle) SBOM storage

Problem

The current SBOMGraph model (normalize/sbom_graph.go) stores all artifacts
in one shared node/edge map, keyed by component ID (PURL). MergeGraph uses
"last write wins" semantics on a component's outgoing edges — see the comment
in MergeGraph:

Whichever scan most recently touched a given node's declared children is
authoritative for that node's entire edge set from here on

This means two artifacts sharing a component (e.g. circl@v1.6.3) cannot
disagree
about that component's transitive dependencies. A rescan of one
artifact silently rewrites the dependency tree seen by every other artifact
that shares the node.

Proposed solution

Replace the mutable (component_id, dependency_id) edge table with a content-addressed
one. Components (PURL, license, name, etc.) are unchanged. An edge points at
the hash of the child's entire subtree (direct_dependency_subtree_hash),
and a component's own subtree_hash is hash(component_id + all children's subtree hashes). Two artifacts that disagree about a shared component's
children produce different subtree_hash values rooted at that component,
so both edge sets coexist as separate rows; where they agree, the rows are
identical and collide on the primary key, so nothing is duplicated. This deduplication works across a whole devguard instance - never storing information twice

This builds on the existing SBOMMerkleEdge model added in
14bcc64.

Schema

-- existing, reused as-is
CREATE TABLE sbom_merkle_edge (
  subtree_hash                  TEXT NOT NULL,
  component_id                  TEXT NOT NULL, -- PURL, or 'ROOT'
  direct_dependency_subtree_hash TEXT NOT NULL,
  PRIMARY KEY (subtree_hash, component_id, direct_dependency_subtree_hash)
);

-- new: index for upward (vuln -> affected artifacts) traversal
CREATE INDEX idx_merkle_edge_child
  ON sbom_merkle_edge (direct_dependency_subtree_hash);

-- new: one row per asset version + SBOM origin (replaces synthetic
-- artifact:/infosource: nodes)
CREATE TABLE asset_version_sbom_source (
  asset_id           TEXT NOT NULL,
  asset_version_name TEXT NOT NULL,
  origin             TEXT NOT NULL, -- e.g. "sbom:package-lock.json"
  root_subtree_hash  TEXT NOT NULL,
  updated_at         TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (asset_id, asset_version_name, origin),
  FOREIGN KEY (asset_version_name, asset_id) REFERENCES asset_versions (name, asset_id)
);

Component metadata (license, name, etc.) stays in the existing components
table, keyed by PURL — unchanged, no duplication.

Core queries

Ingest a new/updated SBOM source:

  1. Build in-memory graph for the source (existing normalize build logic,
    unchanged).
  2. Compute subtree_hash bottom-up (existing computeSubtreeHashes).
  3. INSERT ... ON CONFLICT DO NOTHING all edges — new subtrees insert,
    already-seen subtrees no-op.
  4. Upsert artifact_sbom_source.root_subtree_hash for (artifact, source).

Which artifacts contain vulnerable PURL X (upward walk):

seed:   rows WHERE component_id = X
walk:   recursive join on direct_dependency_subtree_hash = subtree_hash
        (memoize visited subtree_hash to bound fan-in blowup)
stop:   component_id = 'ROOT'
result: join stopped subtree_hash against artifact_sbom_source.root_subtree_hash

Which components does artifact A depend on (downward walk):
Recursive CTE starting at artifact_sbom_source.root_subtree_hash, joining
subtree_hash -> direct_dependency_subtree_hash downward.

Advantages

  • artifacts can say something different about there subtrees - which is actually more common than we thought: even in devguard we are pathing transitive dependencies, package manager work with min version algorithms to find matching subsets of packages to install
  • deduplication across a whole devguard instance
  • an SBOM can be represented as single hash. Since pretty much all devguard developers love hashes I am sure we will come up with cool ideas what todo with them.

Downsides

We need to implement some garbage collection to remove unreferenced subtrees. I cannot come up with a good way to identify those besides some mark and sweep algorithm.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with normalize/sbom_graph.go, the existing SBOMMerkleEdge model, and computeSubtreeHashes to understand the current graph and hash flow. Then trace SBOM ingestion and the existing artifact/version persistence before evaluating the proposed schema, source mapping, and recursive traversal queries. Done means conflicting component subtrees coexist without duplication, sources resolve to root hashes, and both upward vulnerability and downward dependency lookups work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.