bug(llma): MCP-created evaluations never fire because empty conditions are silently rejected
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Bug
LLM analytics evaluations created via the MCP evaluation-create tool never fire on \$ai_generation events, even when enabled: true and the underlying eval logic is correct.
Repro
- Create an evaluation via
mcp__posthog__evaluation-create(ormcp__posthog-local-testing__evaluation-create) with any valid hog/llm_judge config andenabled: true - Send
\$ai_generationevents to the same team - Query
\$ai_evaluationevents — the new eval never appears in the results
Same evaluation created via the UI (/llm-analytics/evaluations → New) fires correctly against the same generations.
Root cause
The evaluation scheduler matcher in nodejs/src/evaluation-scheduler/evaluation-scheduler.ts:184 rejects any evaluation with empty conditions:
```ts
const conditions = evaluation.conditions as EvaluationConditionSet[]
if (conditions.length === 0) {
return { matched: false, reason: 'no_conditions' }
}
```
The UI always sends a default trivial-match condition on create — see products/llm_analytics/frontend/evaluations/llmEvaluationLogic.ts:388 and :464:
```ts
conditions: [{ id: `cond-${Date.now()}`, rollout_percentage: 0, properties: [] }]
```
But the MCP evaluation-create tool at services/mcp/src/tools/llmAnalytics/evaluations/create.ts doesn't expose conditions in its Zod schema, doesn't inject a default, and the model defaults conditions to [] (products/llm_analytics/backend/models/evaluations.py:35):
```python
conditions = models.JSONField(default=list)
```
Result: every MCP-created eval has conditions: [] in Postgres → matcher returns no_conditions → eval is silently dropped → 0 results, no error, no warning.
Why this is bad
- Silent failure: no error returned to MCP caller, no log warning, no UI hint that the eval is dead.
- Inconsistent contract: API model says
conditionsdefaults to[]and is technically valid, but the runtime treats[]as broken. - Blocks any agentic eval workflow: every agent that creates evals via MCP produces non-functional evals.
Recommended fix
Option B (preferred): backend serializer/model defaults a trivial always-match condition.
In products/llm_analytics/backend/models/evaluations.py, in Evaluation.save(), before the existing condition compile loop (line ~78), inject a default condition when self.conditions is empty:
```python
import uuid
...
if not self.conditions:
self.conditions = [{
"id": f"cond-default-{uuid.uuid4().hex[:13]}",
"properties": [],
"rollout_percentage": 100,
}]
```
The existing compile_filters_bytecode(filters, self.team) loop at line 80 will then compile the empty properties: [] into the trivial `["_H", 1, 29]` always-true bytecode, matching what UI-created evals look like in Postgres today.
Why backend over MCP-only fix: this also closes the same hole for direct API callers, tests, and any future SDK that creates evals.
Acceptance criteria
- Creating an evaluation via MCP `evaluation-create` with no `conditions` field results in a row with one default condition (matching UI shape)
- That eval fires on subsequent `$ai_generation` events for the team
- Existing UI flow unaffected (UI still sends its own condition; backend just doesn't override it when present)
- Backfill or migration for existing broken evals optional (they're easily fixed by re-saving)
- Unit test in `products/llm_analytics/backend/models/test/test_evaluations.py` covering: `Evaluation(conditions=[]).save()` → `len(conditions) == 1` and the bytecode compiles to the always-true form
- Optional: add `conditions` as an optional parameter to the MCP `evaluation-create` Zod schema for callers that want to specify properties/rollout
File pointers
- Matcher (the silent reject): `nodejs/src/evaluation-scheduler/evaluation-scheduler.ts:178-205`
- UI default condition: `products/llm_analytics/frontend/evaluations/llmEvaluationLogic.ts:388,464`
- MCP create tool (missing conditions): `services/mcp/src/tools/llmAnalytics/evaluations/create.ts`
- Model save (recommended fix site): `products/llm_analytics/backend/models/evaluations.py:56-88`
- Existing condition compile: `compile_filters_bytecode` from `posthog/cdp/filters.py`
- Existing tests: `products/llm_analytics/backend/models/test/test_evaluations.py`
How I confirmed
In a clean local dev env with 7 enabled evals on team_id=1 (2 UI-created, 5 MCP-created via `mcp__posthog-local-testing__evaluation-create`), I sent 22 `$ai_generation` events from the demo data generator. Direct Postgres query confirmed all 5 MCP-created evals had `conditions = '[]'`. The 2 UI-created evals had `conditions = '[{"id":"cond-...","bytecode":["_H",1,29],"properties":[],"bytecode_error":null,"rollout_percentage":100}]'`. Querying `$ai_evaluation` events from the resulting batch showed only the 2 UI-created evals fired (22 + 21 results). Confirmed restarting the nodejs plugin server (which runs the scheduler) didn't change anything — the bug is at the matcher level, not a cache bug.
LLM context
Filed by an agent (Claude Code) after debugging why MCP-created evals weren't firing in local dev. The fix is small (~5 lines + a test), but the diagnosis took a while because the failure is silent across multiple layers (no logs, no warnings, no UI indicator). Posting here so another agent can pick it up directly.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in products/llm_analytics/backend/models/evaluations.py around Evaluation.save() and read the condition compilation path. Run the existing tests in products/llm_analytics/backend/models/test/test_evaluations.py, then add coverage for saving an evaluation with empty conditions. Done means the saved evaluation has one default condition with always-true bytecode and existing UI-created conditions remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, typescript
- Domain
- api, backend, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100