l3montree-dev / l3montree-dev/devguard
Implement SBOM Merkle tree for minimal storage requirement of component dependencies table
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 161
- Forks
- 43
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 37
Description
Proposal: Content-addressed (Merkle) SBOM storage
Problem
The current SBOMGraph model (normalize/sbom_graph.go) stores all artifacts
in one shared node/edge map, keyed by component ID (PURL). MergeGraph uses
"last write wins" semantics on a component's outgoing edges — see the comment
in MergeGraph:
Whichever scan most recently touched a given node's declared children is
authoritative for that node's entire edge set from here on
This means two artifacts sharing a component (e.g. circl@v1.6.3) cannot
disagree about that component's transitive dependencies. A rescan of one
artifact silently rewrites the dependency tree seen by every other artifact
that shares the node.
Proposed solution
Replace the mutable (component_id, dependency_id) edge table with a content-addressed
one. Components (PURL, license, name, etc.) are unchanged. An edge points at
the hash of the child's entire subtree (direct_dependency_subtree_hash),
and a component's own subtree_hash is hash(component_id + all children's subtree hashes). Two artifacts that disagree about a shared component's
children produce different subtree_hash values rooted at that component,
so both edge sets coexist as separate rows; where they agree, the rows are
identical and collide on the primary key, so nothing is duplicated. This deduplication works across a whole devguard instance - never storing information twice
This builds on the existing SBOMMerkleEdge model added in
14bcc64.
Schema
-- existing, reused as-is
CREATE TABLE sbom_merkle_edge (
subtree_hash TEXT NOT NULL,
component_id TEXT NOT NULL, -- PURL, or 'ROOT'
direct_dependency_subtree_hash TEXT NOT NULL,
PRIMARY KEY (subtree_hash, component_id, direct_dependency_subtree_hash)
);
-- new: index for upward (vuln -> affected artifacts) traversal
CREATE INDEX idx_merkle_edge_child
ON sbom_merkle_edge (direct_dependency_subtree_hash);
-- new: one row per asset version + SBOM origin (replaces synthetic
-- artifact:/infosource: nodes)
CREATE TABLE asset_version_sbom_source (
asset_id TEXT NOT NULL,
asset_version_name TEXT NOT NULL,
origin TEXT NOT NULL, -- e.g. "sbom:package-lock.json"
root_subtree_hash TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (asset_id, asset_version_name, origin),
FOREIGN KEY (asset_version_name, asset_id) REFERENCES asset_versions (name, asset_id)
);
Component metadata (license, name, etc.) stays in the existing components
table, keyed by PURL — unchanged, no duplication.
Core queries
Ingest a new/updated SBOM source:
- Build in-memory graph for the source (existing
normalizebuild logic,
unchanged). - Compute
subtree_hashbottom-up (existingcomputeSubtreeHashes). INSERT ... ON CONFLICT DO NOTHINGall edges — new subtrees insert,
already-seen subtrees no-op.- Upsert
artifact_sbom_source.root_subtree_hashfor(artifact, source).
Which artifacts contain vulnerable PURL X (upward walk):
seed: rows WHERE component_id = X
walk: recursive join on direct_dependency_subtree_hash = subtree_hash
(memoize visited subtree_hash to bound fan-in blowup)
stop: component_id = 'ROOT'
result: join stopped subtree_hash against artifact_sbom_source.root_subtree_hash
Which components does artifact A depend on (downward walk):
Recursive CTE starting at artifact_sbom_source.root_subtree_hash, joining
subtree_hash -> direct_dependency_subtree_hash downward.
Advantages
- artifacts can say something different about there subtrees - which is actually more common than we thought: even in devguard we are pathing transitive dependencies, package manager work with min version algorithms to find matching subsets of packages to install
- deduplication across a whole devguard instance
- an SBOM can be represented as single hash. Since pretty much all devguard developers love hashes I am sure we will come up with cool ideas what todo with them.
Downsides
We need to implement some garbage collection to remove unreferenced subtrees. I cannot come up with a good way to identify those besides some mark and sweep algorithm.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with normalize/sbom_graph.go, the existing SBOMMerkleEdge model, and computeSubtreeHashes to understand the current graph and hash flow. Then trace SBOM ingestion and the existing artifact/version persistence before evaluating the proposed schema, source mapping, and recursive traversal queries. Done means conflicting component subtrees coexist without duplication, sources resolve to root hashes, and both upward vulnerability and downward dependency lookups work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100