clearlydefined / clearlydefined/crawler

Break up "no revision" package fetching

Open
#56 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
60
Forks
38
Avg merge
1h 49m
Merged PRs (30d)
1

Description

When a request to process a package does not have a revision, that defeats the store cache lookup because we don't know the revision so cannot create a proper store key.

Fixing this inline would complicate the fetch code since the store cache look up is generic (in Dispatcher). It would have to prequery the fetcher to build the URN (or get the revision) and then do the cache check.

Unfortunately, the "get latest" work involves getting data that is otherwise useful in processing and somewhat expensive (e.g., calling npmjs.com). So we'd want to keep it around. we'd have to find a place to stuff it on the request and then look for it later. lots of special code.

Alternatively, we could break the fetch/process for non-versioned packages into two steps. The first would resolve the latest version and the queue a new request with the right version and, perhaps, a payload that includes the interesting information it found. The second would be processing as normal. If the payload had the data then that would be an optimization.

This could also get complicated as some fetchers actually fetch multiple things. So if there is a payload, the generic fetch code will skip calling the real fetcher.

Yuck

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.