dgtlmoon / dgtlmoon/changedetection.io

Support short-lived API tokens (OAuth2 client credentials) so watches can use official vendor APIs

Open
#4,344 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.1k
Forks
2k
Avg merge
22h 52m
Merged PRs (30d)
70

Description

### Version

0.55.8 (Docker, `ghcr.io/dgtlmoon/changedetection.io:latest`)

### Context

A growing number of retail sites refuse automated document fetches outright, so the sanctioned way to monitor them is the site's own API. changedetection.io can already watch a JSON endpoint and send custom request headers, but it has no way to obtain or refresh a short-lived API token — which is what nearly every commercial API requires. That gap is what this issue is about.

Two sites I hit this week, tested from a residential IP and a VPN exit, with both fetch backends:

**thredup.com** — Cloudflare Bot Management (`server: cloudflare`, `__cf_bm` cookie):

| Fetcher | Exit IP | Result |
|---|---|---|
| `html_requests` | residential | 403 |
| `html_webdriver` (Playwright) | residential | 403 |
| `html_webdriver` (Playwright) | datacenter VPN | 403 |

**ebay.com** — eBay's own edge (`server: ebay-proxy-server`):

| Request | Result |
|---|---|
| `HEAD /itm/` | **200** |
| `GET /itm/`, plain client | 403 |
| `GET /itm/`, full browser headers | 403 |
| `GET /itm/`, real Chrome via Playwright | 403 |
| `GET /itm/`, VPN exit | 403 |

(Tracking parameters stripped from the eBay URL made no difference.) The HEAD/GET split is worth noting: the IP is not blocklisted, the document fetch specifically is refused.

I'm not asking for a bypass — #1619 and #1700 covered that ground and were closed. The point is that for these sites there is a legitimate path, and changedetection.io is one small feature away from being able to use it.

### The gap

eBay's Browse API `getItem` returns price and availability as JSON, free with a developer account. Watching it in changedetection.io would need:

```
Authorization: Bearer
```

Request headers already support Jinja2 templating (`forms.py:1060`), so a static value works — but eBay's client-credentials token expires in roughly two hours. Jinja2 can render a token, it cannot go fetch one. So an API-backed watch dies within hours of being created, and there's no supported way to keep it alive.

Same shape for most other commercial APIs: OAuth2 client credentials, short-lived bearer token.

### Proposed solution

An optional credential source, configured once and referenced from a watch's headers:

- Config carries: token endpoint URL, client ID, client secret, optional scope, grant type (`client_credentials` first — it's the machine-to-machine case and needs no user interaction)
- changedetection.io requests a token, caches it, and refreshes it when it expires or when a request returns 401
- The current token is exposed to the existing header templating, e.g. `Authorization: Bearer {{ credential('ebay') }}`
- Secrets stored like other credentials in the datastore and redacted in the UI and logs

That reuses the JSON-watch and header machinery already present; the only new part is token lifecycle.

Lighter alternatives if that's more than you want to own:

1. **A header pre-request hook** — let a watch shell out to a command or hit a URL, and bind the result into the header template. More general, less code, pushes the OAuth details onto the user.
2. **Document the pattern instead** — a wiki recipe for an external token-refresher writing to a file that a watch reads. Solves nothing in-app but at least names the workaround.

I'd favour the first-class version: "monitor a price via the vendor's official API" seems like a use case that will only become more common as scraping paths keep closing, and it's a use case no site operator objects to.

Happy to work on a PR if the direction is agreeable — I'd rather check the approach here first than guess at it.

### Environment

- changedetection.io 0.55.8 in Docker
- Playwright fetcher via `dgtlmoon/sockpuppetbrowser`
- Also reproduced with the plain requests fetcher

Related: #4343 (error message for origin-side bot blocks — same investigation, different ask)

Contributor guide

Open the contributing guide

Research direction

Start with the existing Jinja2 header templating path at forms.py:1060 and trace how JSON-watch headers and datastore credentials are handled. Clarify the token endpoint, client-credentials configuration, caching and refresh behavior, and the UI/log redaction requirements; done means an API-backed watch can use a refreshed bearer token without manual updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, authentication, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.