WordPress / WordPress/php-ai-client
Model resolution failures name the requested capability, never the option that actually failed
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 308
- Forks
- 84
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 2
Description
When model resolution fails because of an option, the error names the capability instead — which is usually the one thing that was fine.
What happens
ModelRequirements::fromPromptData() turns a non-null web search config into a required option (src/Providers/Models/DTO/ModelRequirements.php:409-411):
if ($modelConfig->getWebSearch() !== null) {
$requiredOptions[] = new RequiredOption(OptionEnum::webSearch(), true);
}
When no model satisfies that, ModelResolver::resolve() builds its message from the primary capability alone (src/Providers/ModelResolver.php:230-280), producing:
No models found for provider "acme" that support text_generation.
Text generation was supported by every model the provider offers. An option eliminated all of them. The message points at the wrong thing, and the reader's natural next step — checking whether the provider does text generation — confirms that it does, which makes the message actively misleading rather than merely unhelpful. This cost us a meaningful chunk of a debugging session.
The same applies to any required option, not just web search; inputModalities is on every prompt request, so it is the most likely one to surprise someone.
The groundwork already exists and is unused
ModelRequirements::getUnmetRequirements() was added at @since n.e.x.t with this docblock:
Unlike
areMetBy(), this method reports the specific capabilities and options that are unsupported, so that calling code can explain why a model is unsuitable.
ModelResolver's failure path never calls it. This issue is really about finishing that thought.
Proposal
On an empty candidate map, re-run the lookup with the required capabilities only:
- if that also returns nothing, the capability genuinely is the cause and the message stays exactly as it is today;
- if it returns models, intersect
getUnmetRequirements()['options']across them and name the options that every otherwise-suitable model fails to support.
Result:
No models found for provider "acme" that support text_generation. The following requested option is not supported by any of those models: webSearch.
The extra registry lookup happens only on a path that is already throwing, so it costs nothing in the normal case.
Note for whoever reviews
One existing test needs its expectation widened rather than its behaviour changed: PromptBuilderTest::testGenerateResultWithProviderNoModelsThrowsException pins the registry mock to exactly one lookup, and since fromPromptData() always adds inputModalities as a required option, the diagnostic fires and looks a second time. The assertion becomes exactly(2); the thrown message is unchanged.
I have this implemented against trunk with unit tests covering the three cases (option named, capability genuinely unsupported so no option is blamed, several options named).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ModelRequirements::getUnmetRequirements() in src/Providers/Models/DTO/ModelRequirements.php and the failure path in src/Providers/ModelResolver.php. Run the resolver tests and inspect PromptBuilderTest::testGenerateResultWithProviderNoModelsThrowsException; done means option failures identify the unsupported options while genuine capability failures retain the existing message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100