Bug report: PostHog AI reports success creating insights when API actually returned validation errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 7h 27m
- Merged PRs (30d)
- 222
Description
Bug Description
Please describe.
Max AI attempted to create multiple insights for a user, initially reported a problem creating the insights, but later said they exist and gave the user insight names and links to insights that are not found. It looks like the insights were never actually created.
I noticed the insight payloads look to be missing the required query wrapper. The AI was sending:
{
"kind": "TrendsQuery",
"series": [...],
...
}
But the API expects:
{
"query": {
"kind": "TrendsQuery",
"series": [...],
...
}
}
This resulted in a PydanticOutputParserException with "Field required" at location ["query"]. From cd524f8a-3a7e-4476-ac2c-bb082496810b:
"PydanticOutputParserException: [{\"type\":\"missing\",\"loc\":[\"query\"],\"msg\":\"Field required\",\"input\":{\"kind\":\"TrendsQuery\",\"series\":[{\"kind\":\"EventsNode\",\"event\":\"$exception\",\"math\":\"total\",\"properties\":[{\"type\":\"event\",\"key\":\"$pathname\",\"operator\":\"exact\",\"value\":\"/signup\"}]}],\"dateRange\":{\"date_from\":\"-30d\",\"date_to\":null},\"interval\":\"day\",\"trendsFilter\":{\"display\":\"ActionsLineGraph\",\"showLegend\":false,\"showValuesOnSeries\":false,\"yAxisScaleType\":\"linear\",\"aggregationAxisFormat\":\"numeric\"},\"filterTestAccounts\":true}}], occurred at:\n```\n{\"kind\": \"TrendsQuery\", \"series\": [{\"kind\": \"EventsNode\", \"event\": \"$exception\", \"math\": \"total\", \"properties\": [{\"type\": \"event\", \"key\": \"$pathname\", \"operator\": \"exact\", \"value\": \"/signup\"}]}], \"dateRange\": {\"date_from\": \"-30d\", \"date_to\": null}, \"interval\": \"day\", \"trendsFilter\": {\"display\": \"ActionsLineGraph\", \"showLegend\": false, \"showValuesOnSeries\": false, \"yAxisScaleType\": \"linear\", \"aggregationAxisFormat\": \"numeric\"}, \"filterTestAccounts\": true}\n```"
Session Traces: https://us.posthog.com/project/2/llm-analytics/sessions/0d83044a-0c16-4653-be88-bdf2ef9e3efe?date_from=-24h
How to reproduce
- Ask Max to create one or more insights (e.g., "Create a trends insight showing $exception events on /signup over the last 30 days")
- Look for the insights that were "created", note the insights don't exist
- Check trace to see Pydantic validation error
Additional context
From: https://posthoghelp.zendesk.com/agent/tickets/46578 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/49129)
Debug info
Session: https://us.posthog.com/project/sTMFPsFhdP1Ssg/replay/019b9eb3-e1ec-7ac7-8b06-6227238e6b06?t=849
Admin: http://go/adminOrgUS/019aea12-cc53-0000-3f9b-d486307dd1e1 (project ID 260343)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing Max's insight-creation request from the reproduction scenario through the API response and validation-error handling. Compare the payload shape with the required query wrapper, then verify that failed creations are not reported as successful. Add or update coverage for the Pydantic validation-error case and run the relevant tests once the entry point is located.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100