Blog - Why and how we built Stamphog

Summary

We built Stamphog, it's an AI reviewer bot. Shows we are doing cool stuff with AI and help people ship fast. Here's its story.

Headline options

(I don't like any of these so far)

Human reviews aren't always needed in the agent of AI
Fewer human pull requests reviews
Do you really need another human to review your PRs?
Are humans really required to review pull requests?
RIP human pull requests?
Does another human really need to review your pull request?

What (if any) keywords are we targeting?

🤷

Outline (optional)

• We require reviews to merge to production.
• This leads to a lot of "stamp please" because of small teams, no enough reviews, and a lot of changes. Without this, velocity dies.
• Our philosophy about reviews and how that has changed in the age of AI
  • Pre-AI, would we have shipped engineer-writes / engineer-self-reviews / engineer-merges? No.
  • What changed is that the engineer is now the reviewer, and AI writes the code. A second AI doing the sanity check is symmetric.
  • We also have a bunch of agents reviewing code. They are often better at security anyways
  • Small reviews are really sanity checks
• We built Stamphog
  • So to help increase velocity we built Stamphog an LLM reviewer bot engineers can self-request a "stamp" review from. Here's how it works:
  • Add label
  • Determinstic gate
  • LLM review to approve or leave comment
  • Dogfooding LLM analytics
  • Claude Agent SDK + GitHub Actions
• Security
  • The team cared a lot more about the security of this than the functionality. Security incident earlier showed this.
  • Base branch checkout, not PR. Otherwise PR could rewrite its own approval logic.
  • A user could open branch A with malicious changes, branch B with benign changes, then PR B → A. Fix: hardcode the base to master.
  • There are files that always need a human review.
  • Stale review handling
• Is it working?
  • 353 PR reviews attempted, ~25 runs per day.
  • 36 unique PR authors — not just one team dogfooding.
  • $59.56 in LLM spend. Call out the cost-per-review (~17¢) so readers can do the math for their own team size.
  • LLM verdicts when the agent ran:
    • APPROVE: 167 (67%)
    • REFUSE: 42 (17%)
  • ESCALATE: 41 (16%)
  • Gates catch ~11% of PRs before the LLM runs (T2-never) — the deterministic tier is doing real work, not just deferring to the LLM.
  • Key takeaway: of PRs the LLM actually reviews, 1 in 3 isn't a clean approve. Show an example PR where Stamphog refused: https://github.com/PostHog/posthog/pull/55201#pullrequestreview-4136494591)
  • Link to PostHog LLM analytics as the source of the numbers — this is also a dogfooding moment.