clearlydefined / clearlydefined/crawler
Break up "no revision" package fetching
- Dominant language
- JavaScript
- Stars
- 60
- Forks
- 38
- Avg merge
- 1h 49m
- Merged PRs (30d)
- 1
Description
When a request to process a package does not have a revision, that defeats the store cache lookup because we don't know the revision so cannot create a proper store key.
Fixing this inline would complicate the fetch code since the store cache look up is generic (in Dispatcher). It would have to prequery the fetcher to build the URN (or get the revision) and then do the cache check.
Unfortunately, the "get latest" work involves getting data that is otherwise useful in processing and somewhat expensive (e.g., calling npmjs.com). So we'd want to keep it around. we'd have to find a place to stuff it on the request and then look for it later. lots of special code.
Alternatively, we could break the fetch/process for non-versioned packages into two steps. The first would resolve the latest version and the queue a new request with the right version and, perhaps, a payload that includes the interesting information it found. The second would be processing as normal. If the payload had the data then that would be an optimization.
This could also get complicated as some fetchers actually fetch multiple things. So if there is a payload, the generic fetch code will skip calling the real fetcher.
Yuck
Contributor guide
Assessment
This issue has not been assessed yet.