Self-driving: implementation agent should reply to PR review threads it addresses
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Filing via the API, so I can't apply the
enhancementlabel myself — please tag per CONTRIBUTING.
Is your feature request related to a problem?
Please describe.
We've had self-driving running against a private monorepo since late July — about 20 PRs from posthog[bot], several merged. The implementation quality is genuinely good; this is about the review loop around it.
The implementation agent never acknowledges review feedback on GitHub. We checked five of its PRs via GET /pulls/{n}/comments and GET /issues/{n}/comments: posthog[bot] has posted zero issue comments and zero threaded replies on any of them. Our automated reviewer leaves inline findings; nothing ever replies to them; the threads stay unresolved.
Two concrete costs:
- Findings get re-raised after they are fixed. With no reply on record saying a thread was addressed, the next review pass re-flags the same thing. Rounds pile up — two of our PRs reached 53 commits in about two days.
- Unresolved threads block merge under our ruleset. A human then walks every thread and resolves it by hand, even when the agent already fixed the code.
There is a second-order effect too: with no reply, a human reading the PR cannot tell whether feedback was seen and rejected, or simply never reached the agent.
Describe the solution you'd like
When the implementation agent changes code in response to review feedback, have it reply on each originating thread with what changed and the commit SHA, then resolve the thread.
Ordering matters and is easy to get backwards: commit -> reply -> push. If it pushes first, the next reviewer pass starts with no replies in its context and re-flags everything.
Describe alternatives you've considered
- Replying by hand. Works, but scales with PR volume and is the part a human adds least value to.
- Having a different agent reply on PostHog's behalf. This is what we actually built — a GitHub Actions dispatcher that watches for review submissions on
posthog[bot]PRs and hands the PR to another coding agent to fix, reply, and resolve. It works, but a second agent is then speaking for yours, and it cannot say authoritatively why the original change was made the way it was. - Treating the PR as fire-and-forget and closing anything that attracts findings. Wasteful — the PRs are usually right, and a closed PR still bills.
Additional context
Related, and arguably the upstream cause: there is no GitHub-side trigger to get the agent to act on a review at all — feedback only reaches it via the inbox / Slack / Desktop chat. Filed separately.
Also related: the agent does not dismiss a REQUEST_CHANGES review once it has addressed every finding. GitHub never auto-dismisses on push (dismiss_stale_reviews_on_push only clears approvals), so an addressed change-request sits looking blocking forever. Filed separately.
Happy to share specifics from our install if useful.
Debug info
- PostHog Cloud (US)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the implementation agent receives GitHub review feedback and creates or updates pull requests; the issue cites GET /pulls/{n}/comments and GET /issues/{n}/comments as the relevant APIs. Done means addressed threads receive a reply describing the change and commit SHA, are resolved, and follow commit -> reply -> push ordering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github
- Domain
- ai-infra-agents, devtools
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100