PostHog / PostHog/posthog

bug(llma): MCP-created evaluations never fire because empty conditions are silently rejected

Open
#53,730 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug

LLM analytics evaluations created via the MCP evaluation-create tool never fire on \$ai_generation events, even when enabled: true and the underlying eval logic is correct.

Repro

  1. Create an evaluation via mcp__posthog__evaluation-create (or mcp__posthog-local-testing__evaluation-create) with any valid hog/llm_judge config and enabled: true
  2. Send \$ai_generation events to the same team
  3. Query \$ai_evaluation events — the new eval never appears in the results

Same evaluation created via the UI (/llm-analytics/evaluations → New) fires correctly against the same generations.

Root cause

The evaluation scheduler matcher in nodejs/src/evaluation-scheduler/evaluation-scheduler.ts:184 rejects any evaluation with empty conditions:

```ts
const conditions = evaluation.conditions as EvaluationConditionSet[]
if (conditions.length === 0) {
return { matched: false, reason: 'no_conditions' }
}
```

The UI always sends a default trivial-match condition on create — see products/llm_analytics/frontend/evaluations/llmEvaluationLogic.ts:388 and :464:

```ts
conditions: [{ id: `cond-${Date.now()}`, rollout_percentage: 0, properties: [] }]
```

But the MCP evaluation-create tool at services/mcp/src/tools/llmAnalytics/evaluations/create.ts doesn't expose conditions in its Zod schema, doesn't inject a default, and the model defaults conditions to [] (products/llm_analytics/backend/models/evaluations.py:35):

```python
conditions = models.JSONField(default=list)
```

Result: every MCP-created eval has conditions: [] in Postgres → matcher returns no_conditions → eval is silently dropped → 0 results, no error, no warning.

Why this is bad

  • Silent failure: no error returned to MCP caller, no log warning, no UI hint that the eval is dead.
  • Inconsistent contract: API model says conditions defaults to [] and is technically valid, but the runtime treats [] as broken.
  • Blocks any agentic eval workflow: every agent that creates evals via MCP produces non-functional evals.

Recommended fix

Option B (preferred): backend serializer/model defaults a trivial always-match condition.

In products/llm_analytics/backend/models/evaluations.py, in Evaluation.save(), before the existing condition compile loop (line ~78), inject a default condition when self.conditions is empty:

```python
import uuid

...

if not self.conditions:
self.conditions = [{
"id": f"cond-default-{uuid.uuid4().hex[:13]}",
"properties": [],
"rollout_percentage": 100,
}]
```

The existing compile_filters_bytecode(filters, self.team) loop at line 80 will then compile the empty properties: [] into the trivial `["_H", 1, 29]` always-true bytecode, matching what UI-created evals look like in Postgres today.

Why backend over MCP-only fix: this also closes the same hole for direct API callers, tests, and any future SDK that creates evals.

Acceptance criteria

  • Creating an evaluation via MCP `evaluation-create` with no `conditions` field results in a row with one default condition (matching UI shape)
  • That eval fires on subsequent `$ai_generation` events for the team
  • Existing UI flow unaffected (UI still sends its own condition; backend just doesn't override it when present)
  • Backfill or migration for existing broken evals optional (they're easily fixed by re-saving)
  • Unit test in `products/llm_analytics/backend/models/test/test_evaluations.py` covering: `Evaluation(conditions=[]).save()` → `len(conditions) == 1` and the bytecode compiles to the always-true form
  • Optional: add `conditions` as an optional parameter to the MCP `evaluation-create` Zod schema for callers that want to specify properties/rollout

File pointers

  • Matcher (the silent reject): `nodejs/src/evaluation-scheduler/evaluation-scheduler.ts:178-205`
  • UI default condition: `products/llm_analytics/frontend/evaluations/llmEvaluationLogic.ts:388,464`
  • MCP create tool (missing conditions): `services/mcp/src/tools/llmAnalytics/evaluations/create.ts`
  • Model save (recommended fix site): `products/llm_analytics/backend/models/evaluations.py:56-88`
  • Existing condition compile: `compile_filters_bytecode` from `posthog/cdp/filters.py`
  • Existing tests: `products/llm_analytics/backend/models/test/test_evaluations.py`

How I confirmed

In a clean local dev env with 7 enabled evals on team_id=1 (2 UI-created, 5 MCP-created via `mcp__posthog-local-testing__evaluation-create`), I sent 22 `$ai_generation` events from the demo data generator. Direct Postgres query confirmed all 5 MCP-created evals had `conditions = '[]'`. The 2 UI-created evals had `conditions = '[{"id":"cond-...","bytecode":["_H",1,29],"properties":[],"bytecode_error":null,"rollout_percentage":100}]'`. Querying `$ai_evaluation` events from the resulting batch showed only the 2 UI-created evals fired (22 + 21 results). Confirmed restarting the nodejs plugin server (which runs the scheduler) didn't change anything — the bug is at the matcher level, not a cache bug.

LLM context

Filed by an agent (Claude Code) after debugging why MCP-created evals weren't firing in local dev. The fix is small (~5 lines + a test), but the diagnosis took a while because the failure is silent across multiple layers (no logs, no warnings, no UI indicator). Posting here so another agent can pick it up directly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in products/llm_analytics/backend/models/evaluations.py around Evaluation.save() and read the condition compilation path. Run the existing tests in products/llm_analytics/backend/models/test/test_evaluations.py, then add coverage for saving an evaluation with empty conditions. Done means the saved evaluation has one default condition with always-true bytecode and existing UI-created conditions remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
api, backend, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.