WordPress / WordPress/php-ai-client

Model resolution failures name the requested capability, never the option that actually failed

Open
#292 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
308
Forks
84
Avg merge
7d 21h
Merged PRs (30d)
2

Description

When model resolution fails because of an option, the error names the capability instead — which is usually the one thing that was fine.

What happens

ModelRequirements::fromPromptData() turns a non-null web search config into a required option (src/Providers/Models/DTO/ModelRequirements.php:409-411):

if ($modelConfig->getWebSearch() !== null) {
    $requiredOptions[] = new RequiredOption(OptionEnum::webSearch(), true);
}

When no model satisfies that, ModelResolver::resolve() builds its message from the primary capability alone (src/Providers/ModelResolver.php:230-280), producing:

No models found for provider "acme" that support text_generation.

Text generation was supported by every model the provider offers. An option eliminated all of them. The message points at the wrong thing, and the reader's natural next step — checking whether the provider does text generation — confirms that it does, which makes the message actively misleading rather than merely unhelpful. This cost us a meaningful chunk of a debugging session.

The same applies to any required option, not just web search; inputModalities is on every prompt request, so it is the most likely one to surprise someone.

The groundwork already exists and is unused

ModelRequirements::getUnmetRequirements() was added at @since n.e.x.t with this docblock:

Unlike areMetBy(), this method reports the specific capabilities and options that are unsupported, so that calling code can explain why a model is unsuitable.

ModelResolver's failure path never calls it. This issue is really about finishing that thought.

Proposal

On an empty candidate map, re-run the lookup with the required capabilities only:

  • if that also returns nothing, the capability genuinely is the cause and the message stays exactly as it is today;
  • if it returns models, intersect getUnmetRequirements()['options'] across them and name the options that every otherwise-suitable model fails to support.

Result:

No models found for provider "acme" that support text_generation. The following requested option is not supported by any of those models: webSearch.

The extra registry lookup happens only on a path that is already throwing, so it costs nothing in the normal case.

Note for whoever reviews

One existing test needs its expectation widened rather than its behaviour changed: PromptBuilderTest::testGenerateResultWithProviderNoModelsThrowsException pins the registry mock to exactly one lookup, and since fromPromptData() always adds inputModalities as a required option, the diagnostic fires and looks a second time. The assertion becomes exactly(2); the thrown message is unchanged.

I have this implemented against trunk with unit tests covering the three cases (option named, capability genuinely unsupported so no option is blamed, several options named).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ModelRequirements::getUnmetRequirements() in src/Providers/Models/DTO/ModelRequirements.php and the failure path in src/Providers/ModelResolver.php. Run the resolver tests and inspect PromptBuilderTest::testGenerateResultWithProviderNoModelsThrowsException; done means option failures identify the unsupported options while genuine capability failures retain the existing message.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.