JupiterBroadcasting / JupiterBroadcasting/jupiterbroadcasting.com

scraping schedule methodology

Open
#215 6 comments 0 reactions 0 assignees View on GitHub
enhancement feedback requested JB - action needed low priority show-scraper
Dominant language
Sass
Stars
103
Forks
49
Avg merge
3h 42m
Merged PRs (30d)
20

Description

Currently scraping is happening twice daily - at midnight and noon UTC ([code here](https://github.com/JupiterBroadcasting/jupiterbroadcasting.com/blob/51a131f79363b2769f1b7343edf8deb6659732e9/.github/workflows/scrape.yml#L7))

🕖 *note: JBT = Pacific Standard Time ; )*

#### Scraping at:
* 0000 UTC = 1700 JBT
* 1200 UTC = 0500 JBT

#### Considering JB shows typically get published...:

* LUP - 1700-2200 JBT
* Office Hours - 0000 JBT
--
* Coder - 0300 or 0530 JBT, mostly...
* LAN - 0300-0900 JBT
* SSH - 0300-0900 JBT

#### The problem
... it seems shows wouldn't get scraped+added to the website for - worst-case scenario - at most...:
* LUP - 12hrs
* Office Hours - 5 hrs
* Coder, LAN, SSH - 12hrs....

#### I Propose..
I propose a tweak of the scraping schedule or methodology to helpfully reduce the delta between `rss publish time` and `hugo publish time`.

#### Questions
* Is there a way to optimize our show pipeline to play nice with publishing to hugo aswell? Perhaps adding an action to CastaBlasta (our internal publishing tool..) to kick off some rss/Fireside scraping?
* is more JB discipline around publishing times helpful?
* other ideas?

@kbondarev

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.