Wizard: the next generation

In the beginning

We ship fast, which means the best way to integrate PostHog changes:

• New init arguments
• New library versions
• New patterns, like automatic page view tracking with instrumentation-client.js in Next

But LLMs are lossy snapshots of a web that no longer exists, which means that agent coding produced crummy, out-of-date solutions.

First swing

The PostHog wizard resolved this problem:

• Modeling integration progress deterministically made sure of consistent and complete integrations
• We could call out to LLMs to patch a developer's existing files, modifying them more quickly and reliably than they could themselves
• Distributed as a CLI via npx, agents could trigger an integration within a larger development task

To accomplish this, the wizard uses some basic context design: for each framework it supports, a specific prompt is fed to an LLM with the integration code it should use. In a single shot, most users are left better than we found them.

Best of all, a project users assume will take a long time is almost instant.

Downsides

This is great for integration: the modification surface is small: a small diff for a small number of files. A moderate investment in writing some prompts yields thousands of integrations delivered in seconds, with >80% of users quite happy with the results.

But for other tasks, like adding events, the approach falls apart. There are so many small details that matter, and even exhaustive prompting can only prevent so many errors. For a framework like Next, with loads of architectural caveats between client and server, the errors only multiply. Sometimes you get lucky, often times you don't.

Next generation

The problem:

No error correction loop.

A linter and type checker provide loads of useful feedback that allows us to detect coding failures before runtime even happens.

A next generation solution would:

• Understand the context of the project: the whole file tree
• Integrate live documentation from trusted, single sources of truth in the context window: no more tight coupling of docs in code
• Select files to edit or create
• Make edits
• Revise edits based on linter and type checking feedback
• Repeat until all selected files have been revised

Prototyping

I took Letta for a test drive, as it has loads of tools for context design.

Loaded into the trunk:

• Next.js getting started docs for app and pages routers
• Source code for posthog-js (web) and posthog-node
• Markdown docs for PostHog and Next.js

With these details and some structured prompts, the agent could perform the basics of an integration and add events, responding to errors and correcting its implementation.

By adding further documentation, like LLM observability or error tracking, we could continue to expand the surface of the wizard's integration muscle, all by relying on existing sources of truth that we can update trivially.

Next steps

• Getting the Letta prototype a bit more robust and trying additional surface like identify() and error tracking
• Comparison to other approaches, like the Claude Code SDK, to see if it provides a comparative advantage on things like file edits and responding to errors

Start with the Letta prototype and its loaded Next.js documentation, posthog-js, posthog-node, and PostHog sources. Review the proposed context, file-edit, lint, and type-checking loop, then compare it with the Claude Code SDK as described. Done would mean a more robust prototype that handles additional surfaces such as identify() and error tracking.