dgtlmoon / dgtlmoon/changedetection.io
Improve "bot-detection" evasion techniques
- Dominant language
- Python
- Stars
- 34.1k
- Forks
- 2k
- Avg merge
- 23h 38m
- Merged PRs (30d)
- 67
Description
I am opening this issue because https://github.com/dgtlmoon/changedetection.io/discussions/1979 was deleted apparently without a final state (rejected/accepted).
Obersvation: Pure changedetection (no paid services, no proxies in place, ...) is more and more incapable of scraping websites. Because sites have the ability to detect that CD is a bot crawling the site. There are potential counter measures out there that need to be evaluated. More details were in the discussion mentioned.
If this feature request will be rejected it automatically means, that users will be inreasingly be forced to use paid services. Which is fine but I would lile to know where this repository is heading to.
Thanks in advance :)
---
**So there are four sides to this**
- The browser side fingerprint (useragent, sec-ua user agent header, and all the other GPU card fingerprinting ettc tcetc) (note - setting just the 'user-agent' is deprecated and replaced by SEC-UA headers and internal JS navigator object in modern browsers!! https://filipvitas.medium.com/how-to-set-user-agent-header-with-puppeteer-js-and-not-fail-28c7a02165da)
- "smart" work arounds for getting around Cloudflare (with some head-ful browser that first grabs the right pass-through-cookies etc)
- The fingerprint of the actual TCP/IP connection, look up JA3 https://github.com/LyleMi/ja3proxy
- Final extra bit - the reputation of your IP address
---
Contributor guide
Research direction
The issue names no repository files, tests, or entry points. Start by reviewing the deleted discussion and the four proposed areas—browser fingerprinting, Cloudflare workarounds, TCP/IP fingerprinting, and IP reputation—to establish whether the project will pursue any of them. Done requires an agreed scope and acceptance criteria before implementation can begin.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100