PolicyEngine / PolicyEngine/policyengine-household-api

Analytics now count the developer portal's synthetic probe as partner traffic (~576 req/day vs ~487 real)

Open
#1,612 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
3
Avg merge
6h 34m
Merged PRs (30d)
8

Description

@anth-volk flagging this for you since you know the analytics layer best — it grew out of the developer-portal work and makes #1548 more urgent.

What happened

The developer portal runs a synthetic latency probe against the household API every 5 minutes (both channels), driven by Vercel Cron. Until last week it hit the unauthenticated /us/calculate_demo. On 2026-07-02 we switched it to the authenticated /us/calculate path using a dedicated M2M application, so the latency it measures matches what partners experience:

  • Auth0 application: portal-latency-probe, client_id M5laaMnHNbCeZ9azRLdPDFOGLbysRIKR

Side effect: every probe request now lands in the calculate analytics as ordinary authenticated traffic.

Scale of the distortion

From the 90-day client inventory (PR #1602, tools/parity/inventory-report.json): the whole real fleet averages ~487 calculate requests/day across all clients. The probe adds 576/day (2 channels × 288 runs) — more than all real traffic combined. Any consumer of /analytics/calculate/requests (including the portal's staff usage panel) now sees:

  • request volume roughly doubled
  • error rates diluted ~2× (the probe rarely errors)
  • resolved_channel mix pulled toward 50/50 (the probe hits current and frontier equally; real partners are almost entirely on current)
  • the probe's canonical payload variables inflating the ?unique=true variable counts by 576/day

Options

  1. Land #1548 (expose + filter client_id on the endpoint). Consumers can then exclude the probe's client_id (and other internal/testing clients) themselves. This is the most general fix and also unblocks the portal's per-partner usage view.
  2. Additionally or alternatively, maintain a server-side exclusion list (or an exclude_client_ids= / internal=false parameter) so fleet-level summaries are clean by default rather than relying on every consumer knowing the probe's client_id.
  3. Tag synthetic traffic at ingest (e.g. flag requests from known internal client_ids) so it can never be conflated, regardless of query shape.

Happy to contribute the #1548 implementation if that is the preferred direction — the columns and indexes already exist per that issue, so it is an endpoint/serialization change only.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with issue #1548 and the /analytics/calculate/requests endpoint, then inspect tools/parity/inventory-report.json for the traffic baseline and the existing client_id columns and indexes. First resolve which exclusion approach is intended; done means synthetic probe traffic no longer distorts the endpoint’s volume, errors, channel mix, or unique-variable counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
analytics, api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.