JupiterBroadcasting / JupiterBroadcasting/jupiterbroadcasting.com
scraping schedule methodology
- Dominant language
- Sass
- Stars
- 103
- Forks
- 49
- Avg merge
- 3h 42m
- Merged PRs (30d)
- 20
Description
Currently scraping is happening twice daily - at midnight and noon UTC ([code here](https://github.com/JupiterBroadcasting/jupiterbroadcasting.com/blob/51a131f79363b2769f1b7343edf8deb6659732e9/.github/workflows/scrape.yml#L7))
🕖 *note: JBT = Pacific Standard Time ; )*
#### Scraping at:
* 0000 UTC = 1700 JBT
* 1200 UTC = 0500 JBT
#### Considering JB shows typically get published...:
* LUP - 1700-2200 JBT
* Office Hours - 0000 JBT
--
* Coder - 0300 or 0530 JBT, mostly...
* LAN - 0300-0900 JBT
* SSH - 0300-0900 JBT
#### The problem
... it seems shows wouldn't get scraped+added to the website for - worst-case scenario - at most...:
* LUP - 12hrs
* Office Hours - 5 hrs
* Coder, LAN, SSH - 12hrs....
#### I Propose..
I propose a tweak of the scraping schedule or methodology to helpfully reduce the delta between `rss publish time` and `hugo publish time`.
#### Questions
* Is there a way to optimize our show pipeline to play nice with publishing to hugo aswell? Perhaps adding an action to CastaBlasta (our internal publishing tool..) to kick off some rss/Fireside scraping?
* is more JB discipline around publishing times helpful?
* other ideas?
@kbondarev
Contributor guide
Assessment
This issue has not been assessed yet.