PolicyEngine / PolicyEngine/policyengine-household-api
Analytics now count the developer portal's synthetic probe as partner traffic (~576 req/day vs ~487 real)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 3
- Avg merge
- 6h 34m
- Merged PRs (30d)
- 8
Description
@anth-volk flagging this for you since you know the analytics layer best — it grew out of the developer-portal work and makes #1548 more urgent.
What happened
The developer portal runs a synthetic latency probe against the household API every 5 minutes (both channels), driven by Vercel Cron. Until last week it hit the unauthenticated /us/calculate_demo. On 2026-07-02 we switched it to the authenticated /us/calculate path using a dedicated M2M application, so the latency it measures matches what partners experience:
- Auth0 application:
portal-latency-probe, client_idM5laaMnHNbCeZ9azRLdPDFOGLbysRIKR
Side effect: every probe request now lands in the calculate analytics as ordinary authenticated traffic.
Scale of the distortion
From the 90-day client inventory (PR #1602, tools/parity/inventory-report.json): the whole real fleet averages ~487 calculate requests/day across all clients. The probe adds 576/day (2 channels × 288 runs) — more than all real traffic combined. Any consumer of /analytics/calculate/requests (including the portal's staff usage panel) now sees:
- request volume roughly doubled
- error rates diluted ~2× (the probe rarely errors)
resolved_channelmix pulled toward 50/50 (the probe hits current and frontier equally; real partners are almost entirely on current)- the probe's canonical payload variables inflating the
?unique=truevariable counts by 576/day
Options
- Land #1548 (expose + filter
client_idon the endpoint). Consumers can then exclude the probe's client_id (and other internal/testing clients) themselves. This is the most general fix and also unblocks the portal's per-partner usage view. - Additionally or alternatively, maintain a server-side exclusion list (or an
exclude_client_ids=/internal=falseparameter) so fleet-level summaries are clean by default rather than relying on every consumer knowing the probe's client_id. - Tag synthetic traffic at ingest (e.g. flag requests from known internal client_ids) so it can never be conflated, regardless of query shape.
Happy to contribute the #1548 implementation if that is the preferred direction — the columns and indexes already exist per that issue, so it is an endpoint/serialization change only.
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with issue #1548 and the /analytics/calculate/requests endpoint, then inspect tools/parity/inventory-report.json for the traffic baseline and the existing client_id columns and indexes. First resolve which exclusion approach is intended; done means synthetic probe traffic no longer distorts the endpoint’s volume, errors, channel mix, or unique-variable counts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- analytics, api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100