clearlydefined / clearlydefined/service
contribute local harvest and curation data back to public services
- Dominant language
- TypeScript
- Stars
- 51
- Forks
- 46
- PR merge metrics
- No merged PRs in 30d
Description
This is an extension to the idea of decentralized harvest and curation model documented in #386.
At SAP, we have own our scanning processes but we're interested in adopting ClearlyDefined tooling and processes into our existing processes. We'd like to separate component detection from harvest/curation of detected components. For detected components, the basic flow looks like below.
- Use ClearlyDefined curated data on the public services if exists.
- If not exists, run through harvest and curation on ClearlyDefined tooling deployed on our internal network. Use same scan tool set as ClearlyDefined uses but data is stored in our own harvest and curation data stores.
- We'd like to contribute our harvested and curated data back to the community if our completed curation does not already exist in the public services. This can be added individually on-demand or as a batch. Not sure which one is better. Some random thoughts here.
- Curated data - this is for components not yet completed curation.
- If curation PR does not exist, create curation PR and replicate our curation data into comments to enter into the public curation process.
- If curation PR already exists but not completed, participate into an existing curation. One option is to add ours into the existing PR as comments for further review.
- Harvested data
- If (harvested data exists and is complete), skip; Should the data be compared to see if they are same?
- else if (harvest is not in the queue or is in the queue but no scan tool did any work yet), replicate over our harvest data;
- else (harvest is partially done, i.e. 1 scan tool is done but not all), some decision needs to be made here. Ex,
- only replicate the data that doesn't exist already for the individual scan tool or
- replicate the entire harvest data to be consistent to the curation from our end.
This approach depends on the possibility of
- Ability to authenticate user for curation/harvest based on github enterprise instead of github.com. Crawling still uses github.com for authentication. Therefore there are 2 types of authentication needed.
- Ability to use same set of supported data store for harvest and curation but the store is located at different places. Ex, use Azure but not the same location as the public services.
Contributor guide
Research direction
Start by reading the decentralized harvest and curation model in issue #386, then map the proposed local-to-public data flow and the two authentication paths described here. Define the decisions around individual versus batch contribution, partial harvest replication, GitHub Enterprise authentication, and separate data-store locations; done means an agreed implementation scope rather than an open-ended design discussion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, github
- Domain
- authentication, backend-api-design, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100