CouncilDataProject / CouncilDataProject/cdp-backend

Allow ability to flag Events as "try to scrape again next time" and do so on the next CRON run

Open
#212 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
25
Forks
28
PR merge metrics
No merged PRs in 30d

Description

### Feature Description

A way to flag single events as always needing to be scraped again the next time the event gather CRON action runs, regardless of the datetime parameters passed in.

### Use Case

We've another layer of need above our need for [processing only part of a video](https://github.com/CouncilDataProject/cdp-backend/issues/211). We know that someone manually adds the timestamps to denote the timestamp range for the part of video a few days _after_ the hearings are uploaded onto the legislature website. It's very conceivable that when we scrape on our CRON schedule, the timestamps wouldn't be there yet.

If I understand correctly, if the timestamps don't exist our options would be to:
- fall back on scraping the whole video, which we don't want as they can be several hours long (dealbreaker)
- skip the video and don't ingest it, but then the next CRON run the video would not get picked up again (dealbreaker)
- ignore the datetime range passed in, and scrape everything every time

### Solution

Since the scraper runs via datetime range and doesn't really understand things in the unit of a single event, the MVP of this could be if any events within a datetime range flag needing to be retried, `get_events` is called for that same datetime range again on the next run.

Ideally, it wouldn't have to re-scrape a whole daterange though and we could provide a function/lambda/callback to run for just the event that needs to be revisited on the next run and it would keep happening until conditions were met for the video to be ingested.

### Alternatives

As mentioned above we can overcome this by not using the datetime parameters passed to `get_events` and scrape everything all the time, and ignore videos that do not have timestamps in the meantime. Scraping everything all the time will probably not be ideal as the number of bills and hearings grow, particularly because our legislature site was literally built in the 90s and hits random errors a lot.

Another idea is we could only programmatically scrape videos that have timestamps within the datetime range, and come back and manually run event gather for the datetime range a few days/weeks after the videos have been added to try again for the timestamps. Also not ideal but could be a backup.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.