clearlydefined / clearlydefined/service

Downstreaming tool improvements to existing component definitions

Open
#569 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
51
Forks
46
PR merge metrics
No merged PRs in 30d

Description

ClearlyDefined makes use of some great open-source scan tools, such as ScanCode Toolkit, Licensee, and Fossology.

When new updates are released for these tools that could increase their coverage, accuracy, or stability, we want to ensure that all CD definitions equally have a chance to benefit from these enhancements.

For instance, the below two cases have 100+ source files that have incorrectly identified licenses due to non-standard license headers.

- https://clearlydefined.io/definitions/git/github/live-clones/hdf5/f3f29dc7df5f3cc41a5e9462d8c415e540cda3d6
Sample license header: https://github.com/live-clones/hdf5/blob/f3f29dc7df5f3cc41a5e9462d8c415e540cda3d6/src/H5FDtest.c

- https://clearlydefined.io/definitions/git/github/jruby/jruby/8a269e34ff3d242ec90fb08061f79c2757a5e157
Sample license header: https://github.com/jruby/jruby/blob/8a269e34ff3d242ec90fb08061f79c2757a5e157/core/src/main/java/org/jruby/embed/bsf/JRubyEngine.java

While a human can make the correct identification, and even proceed to correct each file one-by-one, this is a time consuming task that is better left to the automated tooling to complete.

For both the above cases, new rules for ScanCode can be introduced that would properly identify the license of each source file.

For ScanCode, it is understood that CD tries to follow as close as possible to the latest stable release, and that a "re-run" is started for all definitions (net new and existing) each time a new version is made available in CD. (@pombredanne please let me know if I recalled this incorrectly).

However, it can still take a considerably long time for the effects of these rules to be realized downstream at the CD definition level from the time they are introduced into ScanCode.

A few solutions have previously been discussed that could help to improve this.

1. Receive more automated information on when new stable versions of tools are released so that they can enter production in CD faster (either by notifying CD developers directly or by some automated process that would obtain the latest version and automate the creation of a PR for that change). This would be achieved by automatically monitoring the release cycle of each scan tool.

2. Create an on-demand feature, where for a given CD definition, a user can raise the priority of that definition for a "re-scan" under new tool versions as they become available in CD. This could perhaps be presented as a banner on the definition letting you know the latest available tools have not yet run for that definition. A corresponding button would then bump up the priority for that definition to be scanned under all new tool versions available at that time (what I guess would now be the "current" available toolset). The banner would then return as soon as any of the tooling on the definition falls out of date again.

Please. Your thoughts are welcome!

Brian

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.