clearlydefined / clearlydefined/crawler

add support for a secondary queue

Open
#571 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
60
Forks
38
Avg merge
1h 49m
Merged PRs (30d)
1

Description

### Description

Add support for configuring a secondary queue. When ready to process more requests, the crawler will look for requests in the primary queue. If none found, it will look for requests in the secondary queue.

Tasks:
- [ ] add configuration for two queues identified as primary and secondary
- [ ] update queue processing code to pull requests from the secondary queue when the primary is empty

### Configuration

As an example, the kubernetes config in [crawler.yml](https://github.com/clearlydefined/crawler/blob/master/crawler.yaml#L42-L46) would have the following changes. There are other places requiring updates including other example configs, config defaults in code, and the code processing the configs and queues.

#### Current Configuration

```
- name: CRAWLER_QUEUE_PREFIX
valueFrom:
secretKeyRef:
name: secrets
key: CRAWLER_QUEUE_PREFIX
```

#### Proposed Configuration

```
- name: CRAWLER_QUEUE_PREFIX
valueFrom:
secretKeyRef:
name: secrets
key: CRAWLER_QUEUE_PREFIX

- name: CRAWLER_SECONDARY_QUEUE_PREFIX
valueFrom:
secretKeyRef:
name: secrets
key: CRAWLER_SECONDARY_QUEUE_PREFIX
```

Contributor guide

Open the contributing guide

Research direction

Start with the Kubernetes example in crawler.yaml at lines 42-46, then trace the other example configurations, config defaults, and queue-processing code mentioned in the issue. Confirm how the primary queue is configured and processed before adding the secondary setting. Done means both queues can be configured and requests are taken from the secondary queue only when the primary is empty.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kubernetes
Domain
backend, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.