dgtlmoon / dgtlmoon/changedetection.io

Improve "bot-detection" evasion techniques

Open
#2,198 33 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
34.1k
Forks
2k
Avg merge
23h 38m
Merged PRs (30d)
67

Description

I am opening this issue because https://github.com/dgtlmoon/changedetection.io/discussions/1979 was deleted apparently without a final state (rejected/accepted).

Obersvation: Pure changedetection (no paid services, no proxies in place, ...) is more and more incapable of scraping websites. Because sites have the ability to detect that CD is a bot crawling the site. There are potential counter measures out there that need to be evaluated. More details were in the discussion mentioned.

If this feature request will be rejected it automatically means, that users will be inreasingly be forced to use paid services. Which is fine but I would lile to know where this repository is heading to.

Thanks in advance :)

---
**So there are four sides to this**

- The browser side fingerprint (useragent, sec-ua user agent header, and all the other GPU card fingerprinting ettc tcetc) (note - setting just the 'user-agent' is deprecated and replaced by SEC-UA headers and internal JS navigator object in modern browsers!! https://filipvitas.medium.com/how-to-set-user-agent-header-with-puppeteer-js-and-not-fail-28c7a02165da)
- "smart" work arounds for getting around Cloudflare (with some head-ful browser that first grabs the right pass-through-cookies etc)
- The fingerprint of the actual TCP/IP connection, look up JA3 https://github.com/LyleMi/ja3proxy
- Final extra bit - the reputation of your IP address

---

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files, tests, or entry points. Start by reviewing the deleted discussion and the four proposed areas—browser fingerprinting, Cloudflare workarounds, TCP/IP fingerprinting, and IP reputation—to establish whether the project will pursue any of them. Done requires an agreed scope and acceptance criteria before implementation can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.