PostHog / PostHog/posthog

Self-driving: implementation agent should reply to PR review threads it addresses

Open
#79,131 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Filing via the API, so I can't apply the enhancement label myself — please tag per CONTRIBUTING.

Is your feature request related to a problem?

Please describe.

We've had self-driving running against a private monorepo since late July — about 20 PRs from posthog[bot], several merged. The implementation quality is genuinely good; this is about the review loop around it.

The implementation agent never acknowledges review feedback on GitHub. We checked five of its PRs via GET /pulls/{n}/comments and GET /issues/{n}/comments: posthog[bot] has posted zero issue comments and zero threaded replies on any of them. Our automated reviewer leaves inline findings; nothing ever replies to them; the threads stay unresolved.

Two concrete costs:

  1. Findings get re-raised after they are fixed. With no reply on record saying a thread was addressed, the next review pass re-flags the same thing. Rounds pile up — two of our PRs reached 53 commits in about two days.
  2. Unresolved threads block merge under our ruleset. A human then walks every thread and resolves it by hand, even when the agent already fixed the code.

There is a second-order effect too: with no reply, a human reading the PR cannot tell whether feedback was seen and rejected, or simply never reached the agent.

Describe the solution you'd like

When the implementation agent changes code in response to review feedback, have it reply on each originating thread with what changed and the commit SHA, then resolve the thread.

Ordering matters and is easy to get backwards: commit -> reply -> push. If it pushes first, the next reviewer pass starts with no replies in its context and re-flags everything.

Describe alternatives you've considered

  • Replying by hand. Works, but scales with PR volume and is the part a human adds least value to.
  • Having a different agent reply on PostHog's behalf. This is what we actually built — a GitHub Actions dispatcher that watches for review submissions on posthog[bot] PRs and hands the PR to another coding agent to fix, reply, and resolve. It works, but a second agent is then speaking for yours, and it cannot say authoritatively why the original change was made the way it was.
  • Treating the PR as fire-and-forget and closing anything that attracts findings. Wasteful — the PRs are usually right, and a closed PR still bills.

Additional context

Related, and arguably the upstream cause: there is no GitHub-side trigger to get the agent to act on a review at all — feedback only reaches it via the inbox / Slack / Desktop chat. Filed separately.

Also related: the agent does not dismiss a REQUEST_CHANGES review once it has addressed every finding. GitHub never auto-dismisses on push (dismiss_stale_reviews_on_push only clears approvals), so an addressed change-request sits looking blocking forever. Filed separately.

Happy to share specifics from our install if useful.

Debug info

  • PostHog Cloud (US)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the implementation agent receives GitHub review feedback and creates or updates pull requests; the issue cites GET /pulls/{n}/comments and GET /issues/{n}/comments as the relevant APIs. Done means addressed threads receive a reply describing the change and commit SHA, are resolved, and follow commit -> reply -> push ordering.

Written by the indexing model from the issue text.

Assessment

Tech stack
github
Domain
ai-infra-agents, devtools
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.