WordPress / WordPress/php-ai-client

Non-generative capability for typed judgement models: classification and scoring with probabilities and confidence

Open
#296 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
308
Forks
84
Avg merge
7d 21h
Merged PRs (30d)
2

Description

The client models seven capabilities, six generative plus embeddings. There is a class of model it does not cover: one that takes state plus typed questions and returns a probability, a choice over a defined set, or a position on a defined scale, each with a confidence figure, and generates nothing. TypeSafe's Jev (System One) is the first I have worked with; the shape is general.

Today the only way in is TEXT_GENERATION with a JSON response schema, which means asking a generative model to emit a number, then parsing prose. A judgement model answers the typed question directly and its answer carries a calibrated confidence, which the parsed number does not. Forcing it through the text path throws that away. It also means a WordPress connector for such a model cannot register as ai_provider, because the provider's key is validated against generative operations it does not have.

Embeddings are the precedent: a non-generative capability with its own enum value, model interface, builder and result DTO (CapabilityEnum::EMBEDDING_GENERATION, EmbeddingGenerationModelInterface, EmbeddingBuilder, EmbeddingResult). A judgement capability would follow the same pattern:

  • CapabilityEnum::CLASSIFICATION (or DECISION; naming is yours)
  • a model interface taking state plus a map of typed questions: noul (probability that a statement is true), choice over named options, score over ordered levels
  • a result DTO: per question, the value, a probability distribution where one applies, and a confidence figure
  • a builder entry point alongside generateEmbedding()

Request and response shape, as the live API has it:

// request
{ "state": { "comment": "..." }, "model": "jev-latest",
  "questions": {
    "spam":  { "type": "noul",   "instructions": "Is this comment spam?" },
    "route": { "type": "choice", "instructions": "How should a moderator handle it?",
               "criteria": { "approve": "...", "hold": "...", "trash": "..." } } } }
// response
{ "model": "jev-latest",
  "answers": {
    "spam":  { "noul": 0.03 },
    "route": { "choice": "approve", "confidence": 0.91,
               "probabilities": { "approve": 0.91, "hold": 0.07, "trash": 0.02 } } },
  "usage": { "input_tokens": 312, "output_tokens": 9 } }

I maintain a WordPress connector for Jev (https://github.com/juanlentino/jev-connector) that has this shape working end to end against the live API, plus a consumer that backs the AI plugin's Comment Moderation through its wpai_comment_analysis_result filter (https://github.com/juanlentino/jev-comment-analysis). Happy to draft the interfaces and DTO as a PR if the shape is welcome; I would rather get the naming and placement from you first.

A companion Trac ticket about the Connectors API side (a documented type for AI services the client does not drive) will follow. Jeff Paul suggested filing here after a conversation about where Jev-like integrations should live.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing EmbeddingGenerationModelInterface, EmbeddingBuilder, EmbeddingResult, and CapabilityEnum::EMBEDDING_GENERATION as the stated precedent. Clarify the capability naming, typed-question model interface, result DTO, builder entry point, and provider registration expectations with maintainers before implementation. Done means the agreed design preserves typed answers, distributions where applicable, and confidence without routing through TEXT_GENERATION.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
ai, api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.