clearlydefined / clearlydefined/crawler
add support for a secondary queue
- Dominant language
- JavaScript
- Stars
- 60
- Forks
- 38
- Avg merge
- 1h 49m
- Merged PRs (30d)
- 1
Description
### Description
Add support for configuring a secondary queue. When ready to process more requests, the crawler will look for requests in the primary queue. If none found, it will look for requests in the secondary queue.
Tasks:
- [ ] add configuration for two queues identified as primary and secondary
- [ ] update queue processing code to pull requests from the secondary queue when the primary is empty
### Configuration
As an example, the kubernetes config in [crawler.yml](https://github.com/clearlydefined/crawler/blob/master/crawler.yaml#L42-L46) would have the following changes. There are other places requiring updates including other example configs, config defaults in code, and the code processing the configs and queues.
#### Current Configuration
```
- name: CRAWLER_QUEUE_PREFIX
valueFrom:
secretKeyRef:
name: secrets
key: CRAWLER_QUEUE_PREFIX
```
#### Proposed Configuration
```
- name: CRAWLER_QUEUE_PREFIX
valueFrom:
secretKeyRef:
name: secrets
key: CRAWLER_QUEUE_PREFIX
- name: CRAWLER_SECONDARY_QUEUE_PREFIX
valueFrom:
secretKeyRef:
name: secrets
key: CRAWLER_SECONDARY_QUEUE_PREFIX
```
Contributor guide
Research direction
Start with the Kubernetes example in crawler.yaml at lines 42-46, then trace the other example configurations, config defaults, and queue-processing code mentioned in the issue. Confirm how the primary queue is configured and processed before adding the secondary setting. Done means both queues can be configured and requests are taken from the secondary queue only when the primary is empty.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kubernetes
- Domain
- backend, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100