PostHog / PostHog/posthog

Bug(PostHog AI): chat / PostHog AI doesn't respond to message, doesn't show an error

Open
#58,779 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

AI bug feature/posthog-ai team/posthog-ai
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug Description

When a user asks PostHog AI a complex question with a large dashboard attached as context, the agent can fail silently with no response shown in the chat.

The Temporal activity that runs the chat agent has a 5 minute (300 s) heartbeat timeout, and heartbeats are only emitted when the LangGraph stream yields a chunk. If the root node runs for longer than 300 s without yielding anything, no heartbeat fires and Temporal cancels the activity with asyncio.CancelledError at ~300 s. No FailureMessage is yielded. The user sees no response and no error.

This is amplified by the activity's retry policy (maximum_attempts=3). Each retry replays from the checkpointer, so the input state grows on every attempt. In one observed trace the state grew from ~95 KB to ~285 KB across three retries, all of which were cancelled at exactly 309 s. In a separate trace from the same user the first attempt was cancelled the same way but a later retry succeeded, so the failure is intermittent. (links to traces in ticket below)

How to reproduce

  1. Open PostHog AI in a project that has at least one moderately large dashboard (around 20+ insights).
  2. Attach the dashboard as context and send a multi-part analytical question (several distinct asks combined into one prompt) in the default product_analytics agent mode.
  3. Wait. After about 5 minutes the agent will be silently cancelled, with no reply or error shown in the chat.

Additional context

Raised in: https://posthoghelp.zendesk.com/agent/tickets/58027 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/59923)

I might raise a PR just to stop the silent failure as well, will attach here once it's done.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Temporal activity that runs the chat agent, then trace the LangGraph stream, heartbeat handling, retry policy, and checkpointer behavior described in the issue. Reproduce with a large dashboard and multi-part question; done means a cancellation or exhausted retry produces a visible error in chat instead of failing silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.