LoveShack-Inc / LoveShack-Inc/represent

Crawler sync configuration settings

Open
#13 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

crawler
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Desc
Currently, running `--crawl` will try to crawl the CT Gov site for bills all the way back to 2011. There's basically no reason that we should ever do this more than once for years `!=` the current year. We need a way to update/fetch new data for certain time periods only

I can think of two basic ways to implement this:

### Option 1: Crawl Scheduler

We could have a sync configuration for each datasource, like

```
datasource | last successful sync | sync_interval
```

Then we could set the current year to once a week or something, and then set non-current years to infinity, or null, or something that ensures we only sync them once. When you start the crawler, it'll look for things that haven't been synced within their desired interval (and aren't currently syncing), and sync them. Then sleep for some period of time, wake and check for things that need to be synced, sleep again, etc

### Option 2: CLI args for crawler

Alternatively, you could allow a CLI arg to be passed to the crawler, call it `--earliest-bill-unix-time` and pass it unix time, then when the crawler runs, it should output something like. You might even want to support a range, like `--latest-bill-unix-time`

```shell
$ repp --crawl --earliest-bill-unix-time=1577836800 # that's 1-January-2020@midnight
...
Crawler will only search for bills newer than 1577836800
...
```

Then you'd have to mess with the implementation of the CtGovCrawler to take an optional timestamp. Since the CTGov website is limited, we'd probably still have to do it in yearly chunks, but that's fine

This option would then require that we handle the scheduling outside of the service itself, we can do that with a cronjob on the box it's running on pretty easily

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the `repp --crawl` entry point and the `CtGovCrawler` implementation mentioned in the issue. Compare the scheduler and CLI-argument approaches, then identify the existing crawl flow and its year-based CT Gov limitations. Done means the selected approach can limit or schedule crawl periods without requiring repeated historical fetches.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.