LoveShack-Inc / LoveShack-Inc/represent
Crawler sync configuration settings
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Desc
Currently, running `--crawl` will try to crawl the CT Gov site for bills all the way back to 2011. There's basically no reason that we should ever do this more than once for years `!=` the current year. We need a way to update/fetch new data for certain time periods only
I can think of two basic ways to implement this:
### Option 1: Crawl Scheduler
We could have a sync configuration for each datasource, like
```
datasource | last successful sync | sync_interval
```
Then we could set the current year to once a week or something, and then set non-current years to infinity, or null, or something that ensures we only sync them once. When you start the crawler, it'll look for things that haven't been synced within their desired interval (and aren't currently syncing), and sync them. Then sleep for some period of time, wake and check for things that need to be synced, sleep again, etc
### Option 2: CLI args for crawler
Alternatively, you could allow a CLI arg to be passed to the crawler, call it `--earliest-bill-unix-time` and pass it unix time, then when the crawler runs, it should output something like. You might even want to support a range, like `--latest-bill-unix-time`
```shell
$ repp --crawl --earliest-bill-unix-time=1577836800 # that's 1-January-2020@midnight
...
Crawler will only search for bills newer than 1577836800
...
```
Then you'd have to mess with the implementation of the CtGovCrawler to take an optional timestamp. Since the CTGov website is limited, we'd probably still have to do it in yearly chunks, but that's fine
This option would then require that we handle the scheduling outside of the service itself, we can do that with a cronjob on the box it's running on pretty easily
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the `repp --crawl` entry point and the `CtGovCrawler` implementation mentioned in the issue. Compare the scheduler and CLI-argument approaches, then identify the existing crawl flow and its year-based CT Gov limitations. Done means the selected approach can limit or schedule crawl periods without requiring repeated historical fetches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100