JupiterBroadcasting / JupiterBroadcasting/show-scraper
General things to discuss/improve
- Dominant language
- Python
- Stars
- 5
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Just wanted to keep a general issue that I can put all the different things I think we could do to improve/modify the scraper.py script. Not addressing immediately, so I can focus on getting https://github.com/JupiterBroadcasting/jupiterbroadcasting.com/issues/110 taken care of, but I'd like to address them eventually.
- [ ] replace lists with sets wherever possible
- lists, while definitely more well known, allow duplicates while using `set()` prevents duplicates automatically and is a lot more efficient to check if an item is in the set because it's all hash'ed (`__hash__`).
- So, if we swap out lists for sets it'll ensure we don't have duplicates by default for hosts, guests, tags, etc...
- [ ] use `pathlib.Path` for all the filenames + paths
- `Path` is a more OOP way of handling path'ing, and has some nice functionality to make using paths easier (i.e. there is a `.exists()` function which returns a boolean for if the file/folder exists)
- [ ] https://github.com/JupiterBroadcasting/show-scraper/pull/28
- currently in my head the usernames map is backwards (haven't reached the spot where I'm trying to understand that yet, so maybe it'll make more sense when I do). But there seems to be consistent slugs (ids) on the right and variations of those on the left. If it was visa-versa we could just make a set out of all the variations (list in YAML, but in the config validation it could be converted to a set)
- [ ] implement linting & formatting with pylint & black
- @kbondarev also mentioned mypy for type enforcement (I've heard of it before, but not done it)
- [ ] make code a bit more DRY (Don't Repeat Yourself)
- For example the `parse_hosts_in_ep` and `parse_guests_in_ep` are almost identical, ideally we'd have some type of object to handle an episode and then it'll just automatically parse that information once we pass in the BeautifulSoup contents or call a `.parse()` function...something like that, so we can have internal reference and don't have to duplicate functions to accomplish the same thing
- [ ] going along with DRY, use more OOP for the code (which will let things be more DRY)
- it will also prevent things like the sponsor's text changing because of whichever episode can get added to the SPONSORS global variable first (example below between 2 full scraper runs and running a diff against them)

- [ ] async instead of threads
- the current way that threads are being used (with a threadpool that wraps around more threadpools below it), I believe can actually be an inhibitor because of the overhead to spin up and tear down those pools (definitely not an expert on it, so I could definitely be wrong).
- instead of using threads though, asyncio is supposed to be a lot more processing efficient and sounds like the perfect task for this situation. Learn more about asyncio in python here (it's how I figured out that it probably would work out well for this use case) : https://youtu.be/iG6fr81xHKA
- This task is going to be the biggest overhaul IMO, so I'd prefer to do this one last
Any working on this I'd prefer to do after I finish #19, so I can at least get an MVP of solving the situation for https://github.com/JupiterBroadcasting/jupiterbroadcasting.com/issues/110
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.