Agent V2: clarify compatibility policy for customizable models with declared tool-call capability
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
## Problem
Agent V2 currently applies the same name-based compatibility baseline to both predefined and customizable models.
This seems reasonable for predefined models, where Dify may intentionally maintain a curated quality/reliability baseline beyond raw API capabilities.
For customizable models, however, this can produce a confusing result:
a model may explicitly declare function/tool-calling support through Dify's existing model capability metadata, but still be classified as `Incompatible` solely because its configured label matches one of the predefined model-family patterns.
For example, this class of behavior was reported in #39623 with a customizable OpenAI-compatible Qwen model.
## Current behavior
Agent V2 currently determines compatibility in:
`web/features/agent-v2/agent-detail/configure/model-compatibility.ts`
The compatibility predicate checks `modelItem.label.en_US` against a curated list of model-name patterns.
It does not currently distinguish between:
- `predefined-model`
- `customizable-model`
or use the model's declared function-calling capability for this decision.
This affects the product surface rather than directly blocking the runtime:
- incompatible models are hidden from the default selector view;
- users can explicitly reveal them through **Show incompatible models**;
- an active model may still be selectable after being revealed;
- once selected, it is still shown as **Incompatible**, with the tooltip saying that the model is not available in the current version.
## Existing capability metadata
Dify already exposes both `fetch_from` and `features` in the model contract.
The selector also already has function-call capability handling for model feature filtering.
So the question is not whether Dify should introduce a new capability-detection system.
The question is whether the **predefined-model compatibility baseline is intentionally meant to apply unchanged to customizable models even when those models explicitly declare the technical capability Agent V2 requires.**
## Proposed behavior
If this is not intentional, I suggest separating two concepts.
### Predefined models
Keep the existing curated Agent compatibility baseline unchanged.
Dify can continue deciding which predefined models meet the expected Agent quality/reliability baseline.
### Customizable models
When a customizable model explicitly declares supported function/tool-calling capability, it should not be classified as `Incompatible` solely because its user-configured label matches a predefined-model blacklist pattern.
This would only establish technical eligibility.
It would **not** make the model a Dify-recommended model.
## Suggested models
The existing `Suggested` policy should remain fully curated by Dify.
A customizable model becoming technically eligible should never automatically make it `Suggested`.
In other words:
**technical eligibility != Dify recommendation**
## Scope
The smallest implementation appears to be limited to the model-selection layer:
1. expose the existing `fetch_from` field on the local `ModelSelectorModel` type;
2. preserve the current compatibility behavior for predefined models;
3. use the existing declared function-calling capability when evaluating customizable models;
4. keep `isAgentSuggestedModel()` unchanged;
5. add focused regression tests for predefined and customizable behavior.
The model-list API already carries `fetch_from`, so this should not require a new backend contract or runtime capability-probing mechanism.
## Non-goals
This proposal does not attempt to:
- guarantee that arbitrary self-hosted models work correctly with Agent V2;
- make every tool-calling model Agent-compatible;
- remove Dify's curated predefined-model baseline;
- automatically recommend customizable models;
- change Agent V2 runtime behavior;
- probe model endpoints automatically;
- solve provider-specific vLLM/OpenAI-compatible runtime issues.
Those provider/runtime compatibility problems should remain separate.
## Acceptance cases
Existing predefined-model compatibility and Suggested behavior should remain unchanged.
For customizable models:
- a customizable model with explicitly declared supported function-calling capability should not become `Incompatible` solely because its label matches a predefined-model pattern;
- a customizable model without the required declared capability should remain unsupported;
- a technically eligible customizable model should not automatically become `Suggested`.
## Maintainer question
Is applying the predefined Agent compatibility baseline unchanged to `customizable-model` entries an intentional product policy?
If not, I would like to implement the scoped distinction above and add focused regression tests.
If this direction is acceptable, please let me know or assign the issue to me. I am also happy to adjust the scope before implementation based on the intended Agent V2 model policy.
Contributor guide
Research direction
Start in web/features/agent-v2/agent-detail/configure/model-compatibility.ts and inspect the ModelSelectorModel type, the compatibility predicate, existing function-call capability filtering, and isAgentSuggestedModel(). Add focused regression tests covering predefined models, customizable models with and without declared function-calling capability, and unchanged Suggested behavior; done means the acceptance cases pass without backend or runtime changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, frontend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100