apache / apache/buildstream

[bst-1] unaliased-url annoying w/ source plugins like cpan, pypi, cargo, etc

Open
#1,631 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

These source plugins have a remote package index (crates.io, etc) that they scan to find new versions of dependencies. Then from these package indexes they construct full download URLs, and then download sources from these URLs. From the plugin user's perspective, there is nothing to alias because there are no URL's involved. The user just says "get me package foo from pypi" and the plugin does that.

Buildstream complains that these sources aren't using aliases. Since these sources sometimes download hundreds of libraries for a build, this can end up spamming the build log with hundreds of lines of warnings.

Potential solutions:
- make every plugin provide a way to override the path to the index, and make every project that uses these plugins override these variables with an alias (This is clunky and a bit ugly, IMO. even then this can't work 100% of the time. What about package indexes that return data to map versions -> download urls, or something similar?)
- make a way for projects to opt out of the warning in general. Quick and dirty
- make a way for plugins to opt out of the warning (i.e. "I know this URL I'm giving you won't have an alias; don't raise an error"). Probably the best middle-of-the-road solution IMO

This probably applies to bst-2 also, though I haven't tested it at all

Contributor guide

Open the contributing guide

Research direction

Start by tracing where BuildStream reports unaliased URLs for source plugins, then inspect the cpan, pypi, and cargo plugin flows that resolve package indexes into download URLs. Compare the proposed project-level and plugin-level opt-out approaches; done should mean these legitimate plugin downloads no longer produce hundreds of misleading warnings without weakening alias checks for ordinary URLs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.