dgtlmoon / dgtlmoon/changedetection.io
Support short-lived API tokens (OAuth2 client credentials) so watches can use official vendor APIs
- Dominant language
- Python
- Stars
- 34.1k
- Forks
- 2k
- Avg merge
- 22h 52m
- Merged PRs (30d)
- 70
Description
### Version
0.55.8 (Docker, `ghcr.io/dgtlmoon/changedetection.io:latest`)
### Context
A growing number of retail sites refuse automated document fetches outright, so the sanctioned way to monitor them is the site's own API. changedetection.io can already watch a JSON endpoint and send custom request headers, but it has no way to obtain or refresh a short-lived API token — which is what nearly every commercial API requires. That gap is what this issue is about.
Two sites I hit this week, tested from a residential IP and a VPN exit, with both fetch backends:
**thredup.com** — Cloudflare Bot Management (`server: cloudflare`, `__cf_bm` cookie):
| Fetcher | Exit IP | Result |
|---|---|---|
| `html_requests` | residential | 403 |
| `html_webdriver` (Playwright) | residential | 403 |
| `html_webdriver` (Playwright) | datacenter VPN | 403 |
**ebay.com** — eBay's own edge (`server: ebay-proxy-server`):
| Request | Result |
|---|---|
| `HEAD /itm/` | **200** |
| `GET /itm/`, plain client | 403 |
| `GET /itm/`, full browser headers | 403 |
| `GET /itm/`, real Chrome via Playwright | 403 |
| `GET /itm/`, VPN exit | 403 |
(Tracking parameters stripped from the eBay URL made no difference.) The HEAD/GET split is worth noting: the IP is not blocklisted, the document fetch specifically is refused.
I'm not asking for a bypass — #1619 and #1700 covered that ground and were closed. The point is that for these sites there is a legitimate path, and changedetection.io is one small feature away from being able to use it.
### The gap
eBay's Browse API `getItem` returns price and availability as JSON, free with a developer account. Watching it in changedetection.io would need:
```
Authorization: Bearer
```
Request headers already support Jinja2 templating (`forms.py:1060`), so a static value works — but eBay's client-credentials token expires in roughly two hours. Jinja2 can render a token, it cannot go fetch one. So an API-backed watch dies within hours of being created, and there's no supported way to keep it alive.
Same shape for most other commercial APIs: OAuth2 client credentials, short-lived bearer token.
### Proposed solution
An optional credential source, configured once and referenced from a watch's headers:
- Config carries: token endpoint URL, client ID, client secret, optional scope, grant type (`client_credentials` first — it's the machine-to-machine case and needs no user interaction)
- changedetection.io requests a token, caches it, and refreshes it when it expires or when a request returns 401
- The current token is exposed to the existing header templating, e.g. `Authorization: Bearer {{ credential('ebay') }}`
- Secrets stored like other credentials in the datastore and redacted in the UI and logs
That reuses the JSON-watch and header machinery already present; the only new part is token lifecycle.
Lighter alternatives if that's more than you want to own:
1. **A header pre-request hook** — let a watch shell out to a command or hit a URL, and bind the result into the header template. More general, less code, pushes the OAuth details onto the user.
2. **Document the pattern instead** — a wiki recipe for an external token-refresher writing to a file that a watch reads. Solves nothing in-app but at least names the workaround.
I'd favour the first-class version: "monitor a price via the vendor's official API" seems like a use case that will only become more common as scraping paths keep closing, and it's a use case no site operator objects to.
Happy to work on a PR if the direction is agreeable — I'd rather check the approach here first than guess at it.
### Environment
- changedetection.io 0.55.8 in Docker
- Playwright fetcher via `dgtlmoon/sockpuppetbrowser`
- Also reproduced with the plain requests fetcher
Related: #4343 (error message for origin-side bot blocks — same investigation, different ask)
Contributor guide
Research direction
Start with the existing Jinja2 header templating path at forms.py:1060 and trace how JSON-watch headers and datastore credentials are handled. Clarify the token endpoint, client-credentials configuration, caching and refresh behavior, and the UI/log redaction requirements; done means an API-backed watch can use a refreshed bearer token without manual updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, authentication, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100