luttje / luttje/glua-api-snippets
Optimize scraping
Open
Nobody has claimed this yet.
enhancement
help wanted
low-priority
- Dominant language
- TypeScript
- Stars
- 45
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
Currently we scrape the entire wiki when there's been changes. However it's perfectly achievable to only scrape the changed pages. We could then merge those pages with content of an existing release. This would save GitHub and the Facepunch wiki valuable resources.
I'm making note of this here, but placing it as low-priority for now.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating the scraping entry point and how an existing release is represented; done means detecting changed wiki pages, merging them with existing release content, and avoiding an unnecessary full-wiki scrape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100