Erik-Debye / Erik-Debye/SWA-Scraper
Better solutions needed to escape bot detection
- Dominant language
- JavaScript
- Stars
- 5
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Currently, the program works but definitely isn't perfect and does frequently get detected for being a bot. In particular, I have noticed that once the program gets detected, restarting the program at the point of detection does allow the program to continue.
My guess is that this is due to fingerprinting the chrome profile used, and when the program restarts, puppeteer creates a new temp profile and this escapes detection. Not sure where to start to find a fix.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named, and the issue does not define a specific fix. Start by tracing how Puppeteer creates and reuses the Chrome profile and how the program resumes after detection. Done requires a reproducible mitigation for bot detection with the scraper's existing workflow verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, supabase
- Domain
- backend, web-dev
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100