CouncilDataProject / CouncilDataProject/cdp-backend

Add params to event gather pipeline to allow long-runnable and log errors / skipped events

Open
#195 0 comments 0 reactions 0 assignees View on GitHub
enhancement event gather pipeline
Dominant language
Python
Stars
25
Forks
28
PR merge metrics
No merged PRs in 30d

Description

### Feature Description

_A clear and concise description of the feature you're requesting._

Add parameters:
* `batch-size` an optional integer that will be used to iteratively slice and run the pipeline on that many events at a time. I.e. if the gather for the specified time range finds 50 events but the batch size is 10, the pipeline will run 5 independent times each with 10 events to process.
* `skip-errored-events-during-processing` that will ignore events that raise an error during processing. Enough debug info should be gathered / kept that the log printed out after the pipeline finishes contains the event details and "the thing that errored".
* `skip-errored-events-during-gather` that will ignore events that fail to scrape / gather. Similar to the above parameter, enough debug info should be printed after scraping. "Found 20 events, skipping 2 due to errors" for example.

Also would be really interesting to see if I can allow certain errors. `retry-errors=[ConnectionError]`

### Use Case

_Please provide a use case to help us understand your request in context._

I am backfilling a lot of data for certain instances and it is becoming annoying to process week by week. This is generally required for a couple of reasons:

* storage space on machine (GHA runners only have 16 GB of disk so can't download and process more than ~4 meeting videos at a time) -- hence batch size
* there are errors in less than 1% of events that aren't random connection errors. These are things like the video page being parsed incorrectly and such.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.