clearlydefined / clearlydefined/service

Store newly crawled definitions received on webhook

Open
#515 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
51
Forks
46
PR merge metrics
No merged PRs in 30d

Description

Whereas on the production environment definitions end up shared from the crawler to the service by using Azure Storage, and on file-based local deployments they end up shared because they read/write to the same folder, there is currently no shipping of definitions from the crawler into the service store (the current implementation of definitionService.computeAndStore does not store new definitions, in spite of its name).

My proposal is to make it so definitionService.computeAndStore will store previously unavailable definitions. This would allow local deployments based on mongo to be able to acquire definitions (without manually importing them from files).

Update: Reading the code further, I understand there are checks in place to determine if the definition has been created using a defined tool. Will now check if tooling info is made available by the crawler on the webhook message.

Update 2: On further analysis, this is looking like the crawler does not inform the tools used on described.tools (it does ship described.toolScore though).

### Updated Proposal:
Keep honoring the check and corresponding tests as described in the comment lines following.
```javascript
// If no tools participated in the creation of the definition then don't bother storing.
// Note that curation is a tool so no tools really means there the definition is effectively empty.
```
But, check for the existence of either: an element in 'tools' OR an integer greater than 0 in 'toolScore.total'. Such integer would be a positive indication that a tool has been applied successfully to obtain the definition.

Contributor guide

Open the contributing guide

Research direction

Begin at definitionService.computeAndStore and trace the webhook payload fields described.tools and described.toolScore, along with the existing checks and corresponding tests mentioned in the issue. Done means previously unavailable definitions are stored when tools is non-empty or toolScore.total is a positive integer, while the existing no-tools behavior and tests remain honored.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, typescript
Domain
backend, database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.