aboutcode-org / aboutcode-org/dejacode

BUG: Improve Package from PurlDB failures

Đang mở
#303 6 bình luận 0 reaction 1 người được giao Được @tdruez nhận Xem trên GitHub
bug HighPriority PackageSet
Ngôn ngữ chính
Python
Star
50
Fork
27
Merge trung bình
4 giờ 51 phút
Pull request đã merge (30 ngày)
11

Mô tả

## Extracted from https://github.com/aboutcode-org/dejacode/issues/295#issuecomment-2824782627

> Running "Improve Package from PurlDB" fails with duplicate key value violates unique constraint"component_catalog_packag_dataspace_id_type_namesp_c6620419_uniq"DETAIL:Key(dataspace_id,type,namespace,name,version,qualifiers,subpath,download_url,filename)=(3,npm,,parse-json,4.0.0,,,https://registry.npmjs.org/parse-json/-/parse-json-4.0.0.tgz,parse-json-4.0.0.tgz)alreadyexists. since assigning the download_url would make it a fully duplicate package.

> Attempting to enhance the package in the product with data from PurlDB fails, because assigning the download URL would violate the uniqueness constraint that covers dataspace_id, type, namespace, name, version, qualifiers, subpath, download_url, and filename

This is a corner case where the data pulled from the PurlDB can trigger a "unique constraint" violation when applied to a package.

## Context

PURL from a CycloneDX SBOM: `pkg:npm/parse-json@4.0.0`
Note that in that data source context, no `download_url` is provided.

"There are multiple entries in the PurlDB for this Package." -> 4 results in the PurlDB for this PURL.

https://public.purldb.io/api/packages/?purl=pkg:npm/parse-json@4.0.0

```
[
{
"uuid": "d1eb90f6-5115-49b8-a5f8-782e948cbd3d",
"filename": "parse-json-4.0.0.tgz",
"purl": "pkg:npm/parse-json@4.0.0",
"type": "npm",
"name": "parse-json",
"version": "4.0.0",
"download_url": "https://registry.npmjs.org/parse-json/-/parse-json-4.0.0.tgz",
"sha1": "be35f5425be1f7f6c747184f98a788cb99477ee0",
"package_content": null,
"package_sets": []
},
{
"uuid": "bd3c0519-50d6-47a9-b01a-d3241e0ef641",
"filename": null,
"package_sets": [],
"purl": "pkg:npm/parse-json@4.0.0",
"type": "npm",
"namespace": "",
"name": "parse-json",
"version": "4.0.0",
"download_url": "https://registry.npmjs.com/parse-json/-/parse-json-4.0.0.tgz",
"sha1": "be35f5425be1f7f6c747184f98a788cb99477ee0",
"package_sets": []
},
{
"uuid": "0ff3a534-5570-4289-8560-6e46bb3bad4f",
"filename": "parse-json-4.0.0.tgz",
"purl": "pkg:npm/%40types/parse-json@4.0.0",
"type": "npm",
"namespace": "@types",
"name": "parse-json",
"version": "4.0.0",
"download_url": "https://registry.npmjs.org/@types/parse-json/-/parse-json-4.0.0.tgz",
"package_content": null,
"package_sets": [
{
"uuid": "721da624-9d02-401f-82b6-46142449635d",
"packages": [
"https://public.purldb.io/api/packages/0ff3a534-5570-4289-8560-6e46bb3bad4f/",
"https://public.purldb.io/api/packages/ec74c5e3-676b-489a-a059-4176bdfe6ea8/"
]
}
]
},
{
"uuid": "680f4c25-68b3-4e21-8c16-6db2a822e752",
"filename": null,
"purl": "pkg:npm/%40types/parse-json@4.0.0",
"type": "npm",
"namespace": "@types",
"name": "parse-json",
"version": "4.0.0",
"download_url": "https://registry.npmjs.com/@types/parse-json/-/parse-json-4.0.0.tgz",
"package_content": null,
"package_sets": [
{
"uuid": "c4608309-c909-4a86-bf75-c3363d50bf98",
"packages": [
"https://public.purldb.io/api/packages/680f4c25-68b3-4e21-8c16-6db2a822e752/",
"https://public.purldb.io/api/packages/ec74c5e3-676b-489a-a059-4176bdfe6ea8/"
]
}
],
}
]
```

1. Since the provided purl has no namespace, ie: `pkg:npm/parse-json@4.0.0`, the entries for `pkg:npm/%40types/parse-json@4.0.0` should not be returned
2. The default ordering of the results should be by each PURL field. From `packageurl.contrib.django.models.PackageURLQuerySetMixin.order_by_package_url`:
```
PACKAGE_URL_FIELDS = ("type", "namespace", "name", "version", "qualifiers", "subpath")
def order_by_package_url(self):
"""Order by Package URL fields."""
return self.order_by(*PACKAGE_URL_FIELDS)
```
3. The 2 entries for `pkg:npm/parse-json@4.0.0` are the same, the `sha1` id equal, the only difference is the domain in the `download_url`: `npmjs.org` vs `npmjs.com`. The issue in the DejaCode context using the PurlDB is which one of those package should we use?
An approach could be to take the common fields values from the multiple entries. In this case, as the only know value on the DejaCode side is `pkg:npm/parse-json@4.0.0`, we could ignore the download_url and hash and import everything that is shared across the PurlDB records.

## TODO (Fix) in DejaCode

1. The task should not break, any issues happening during the "Improve Packages from PurlDB" should be properly handled and logged during the `improve_packages_from_purldb` task.

2. Using data pulled from PurlDB onto a package should not raise a unique constraint violation.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.