clearlydefined / clearlydefined/crawler

Proposal: Add crawler for Mariner RPM packages

Open
#490 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
60
Forks
38
Avg merge
1h 49m
Merged PRs (30d)
1

Description

_Related to https://github.com/clearlydefined/clearlydefined/issues/156._

I propose adding the base Mariner 2.0 RPM repositories as a new harvest source. Specifically:
- https://packages.microsoft.com/cbl-mariner/2.0/prod/base/x86_64
- https://packages.microsoft.com/cbl-mariner/2.0/prod/base/aarch64
- https://packages.microsoft.com/cbl-mariner/2.0/prod/base/srpms

These will be used to harvest packages for coordinates of type `rpm` and provider `mariner`.

# Fetcher

A new Mariner fetcher will be added that is configured with these repositories. The fetcher will cache the repositories repomd.xml metadata file and the packages sqlite database file, similar to debian's package file map caching.

On receipt of a request for `cd:rpm/mariner/-/$name/$revision`, the fetcher will query each repository package database file for a match. If it finds a match, it will pull the RPM from the repository, and then extract it using `rpm2cpio` and `cpio`.

_Note that there may be multiple matches, e.g "noarch" RPMs present in both the x86_64 and aarch64 repos. In this case the first one encountered is selected. _

If the RPM is a binary RPM, the source RPM URL is determined and included in the response, so that the clearlydefined service can use this as the source location URL.

# Extracter

A new RPM extracter will be added. This runs the license detection tools over the extracted RPM. for binary RPMs this also determines the coordinates of the source package.

# Considerations for adding a new harvest source

## Discoverability
The packages are organized in a standard yum repository format, so can be discovered using yum/dnf/tdnf or any tool capable of parsing that format. The packages themselves are in RPM format.

## Primary Source
Yes, this is the primary source

## Reputability – is this repository operated by a reputable organization? What is the purpose behind running this repository? Is there an identifiable team that can be reached in the event of any issues?
Microsoft runs package.microsoft.com to host Linux packages. Mariner is our Linux distribution for Azure first-party services. I can find contact details for packages.microsoft.com admins.

## Security – how secure is the repository? Is there a team that is available to handle issues in a timely manner when they arise? How fast do they respond to issues, such as when a security vulnerability is planted as a backdoor in a package?

The external sources are scanned by Mariner prior to building and packaging them.=, and the Mariner team provides CVE fixes for vulnerabilities discovered in packages.

## Automation – does the repository support an API to support pulling of information? If not, is the package index organized in a schematized format that can programmatically queried using the package name and version and queried using HTTP(s). When using HTTP to mine data, ClearlyDefined should check for the existence of robots.txt or robot headers that indicate such mining is unacceptable. How much effort is it to automate the process?

There isn’t an HTTP API. Clients have to download the repository metadata files to discover what packages are available, similar to yum/dnf/tdnf.

## Relationship – reach out to the organization that maintains the repository to indicate that ClearlyDefined wishes to harvest data from their repository, with an explanation on how harvesting is done, what the data is used for and how much additional traffic this could result in. Identify/Resolve any concerns and provide a contact from ClearlyDefined in the event they need to support in case of an issue.
I'll find contact details for packages.microsoft.com maintainers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.