langgenius / langgenius/dify

Agent V2: clarify compatibility policy for customizable models with declared tool-call capability

Open
#42,092 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

## Problem

Agent V2 currently applies the same name-based compatibility baseline to both predefined and customizable models.

This seems reasonable for predefined models, where Dify may intentionally maintain a curated quality/reliability baseline beyond raw API capabilities.

For customizable models, however, this can produce a confusing result:

a model may explicitly declare function/tool-calling support through Dify's existing model capability metadata, but still be classified as `Incompatible` solely because its configured label matches one of the predefined model-family patterns.

For example, this class of behavior was reported in #39623 with a customizable OpenAI-compatible Qwen model.

## Current behavior

Agent V2 currently determines compatibility in:

`web/features/agent-v2/agent-detail/configure/model-compatibility.ts`

The compatibility predicate checks `modelItem.label.en_US` against a curated list of model-name patterns.

It does not currently distinguish between:

- `predefined-model`
- `customizable-model`

or use the model's declared function-calling capability for this decision.

This affects the product surface rather than directly blocking the runtime:

- incompatible models are hidden from the default selector view;
- users can explicitly reveal them through **Show incompatible models**;
- an active model may still be selectable after being revealed;
- once selected, it is still shown as **Incompatible**, with the tooltip saying that the model is not available in the current version.

## Existing capability metadata

Dify already exposes both `fetch_from` and `features` in the model contract.

The selector also already has function-call capability handling for model feature filtering.

So the question is not whether Dify should introduce a new capability-detection system.

The question is whether the **predefined-model compatibility baseline is intentionally meant to apply unchanged to customizable models even when those models explicitly declare the technical capability Agent V2 requires.**

## Proposed behavior

If this is not intentional, I suggest separating two concepts.

### Predefined models

Keep the existing curated Agent compatibility baseline unchanged.

Dify can continue deciding which predefined models meet the expected Agent quality/reliability baseline.

### Customizable models

When a customizable model explicitly declares supported function/tool-calling capability, it should not be classified as `Incompatible` solely because its user-configured label matches a predefined-model blacklist pattern.

This would only establish technical eligibility.

It would **not** make the model a Dify-recommended model.

## Suggested models

The existing `Suggested` policy should remain fully curated by Dify.

A customizable model becoming technically eligible should never automatically make it `Suggested`.

In other words:

**technical eligibility != Dify recommendation**

## Scope

The smallest implementation appears to be limited to the model-selection layer:

1. expose the existing `fetch_from` field on the local `ModelSelectorModel` type;
2. preserve the current compatibility behavior for predefined models;
3. use the existing declared function-calling capability when evaluating customizable models;
4. keep `isAgentSuggestedModel()` unchanged;
5. add focused regression tests for predefined and customizable behavior.

The model-list API already carries `fetch_from`, so this should not require a new backend contract or runtime capability-probing mechanism.

## Non-goals

This proposal does not attempt to:

- guarantee that arbitrary self-hosted models work correctly with Agent V2;
- make every tool-calling model Agent-compatible;
- remove Dify's curated predefined-model baseline;
- automatically recommend customizable models;
- change Agent V2 runtime behavior;
- probe model endpoints automatically;
- solve provider-specific vLLM/OpenAI-compatible runtime issues.

Those provider/runtime compatibility problems should remain separate.

## Acceptance cases

Existing predefined-model compatibility and Suggested behavior should remain unchanged.

For customizable models:

- a customizable model with explicitly declared supported function-calling capability should not become `Incompatible` solely because its label matches a predefined-model pattern;
- a customizable model without the required declared capability should remain unsupported;
- a technically eligible customizable model should not automatically become `Suggested`.

## Maintainer question

Is applying the predefined Agent compatibility baseline unchanged to `customizable-model` entries an intentional product policy?

If not, I would like to implement the scoped distinction above and add focused regression tests.

If this direction is acceptable, please let me know or assign the issue to me. I am also happy to adjust the scope before implementation based on the intended Agent V2 model policy.

Contributor guide

Open the contributing guide

Research direction

Start in web/features/agent-v2/agent-detail/configure/model-compatibility.ts and inspect the ModelSelectorModel type, the compatibility predicate, existing function-call capability filtering, and isAgentSuggestedModel(). Add focused regression tests covering predefined models, customizable models with and without declared function-calling capability, and unchanged Suggested behavior; done means the acceptance cases pass without backend or runtime changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, frontend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.