agentscope-ai / agentscope-ai/QwenPaw
[Bug] Model detection incorrectly classifies Zhipu AI GLM-4.6V-Flash as pure text model instead of multimodal vision model
- Vorherrschende Sprache
- Python
- Sterne
- 34.9k
- Forks
- 3.1k
- Ø Merge
- 1 T. 15 Std.
- Gemergte PRs (30 T.)
- 225
Beschreibung
## Description
When configuring Zhipu AI's `glm-4.6v-flash` model in CoPaw, the model detection incorrectly identifies it as a pure text model instead of a multimodal model, despite it being a vision-capable model.
## Version
v1.0.1
## OS
Windows 11 23H2
## Steps to Reproduce
1. Configure Zhipu AI provider with API key in CoPaw
2. Add model configuration for `glm-4.6v-flash` (or `glm-4v-flash`)
3. Run CoPaw's model detection/probe mechanism
4. Observe the model being classified as pure text model (not multimodal)
5. Check the configuration file at `.copaw.secret/providers/builtin/zhipu-cn.json`
6. Observe `supports_multimodal: false` despite model having vision capabilities
## Expected Behavior
CoPaw should correctly detect Zhipu AI's `glm-4.6v-flash` (and similar models ending with `v` or `Flash`) as multimodal vision models and set:
- `supports_multimodal: true`
- `supports_image_input: true`
- Appropriate `preferred_source` for vision inputs
## Actual Behavior
The model is incorrectly classified as pure text model:
- `supports_multimodal: false`
- `supports_image_input: false`
- No `preferred_source` for vision inputs
- Probe returns text-only capabilities
## Evidence
Configuration file shows incorrect classification:
```json
{
"glm-4.6v-flash": {
"supports_multimodal": false, // ❌ Should be true
"supports_image_input": false, // ❌ Should be true
"preferred_modality": null // ❌ Should have vision options
}
}
```
## Impact
### User Impact
- **High**: Users cannot use image input features with Zhipu AI vision models
- Vision-capable models (GLM-4V series) are incorrectly treated as text-only
- Prevents legitimate use of multimodal capabilities
### Functional Impact
- Image input detection fails
- Vision model features are disabled
- Manual intervention required to fix configuration
## Root Cause Analysis
### Possible Causes
1. **Probe mechanism issue**: The probe test may not properly detect Zhipu AI's vision capabilities
2. **Naming convention gap**: Models with `v` suffix (e.g., `4v`, `4.6v`) may not be recognized as vision models
3. **API response parsing**: Zhipu AI API response format may differ from expected structure
4. **Hardcoded model list**: May rely on hardcoded list without Zhipu vision models
### Suggested Investigation
- Check probe logic in CoPaw's model detection module
- Review Zhipu AI API documentation for vision model detection
- Compare with other providers' vision model detection (e.g., OpenAI's GPT-4V)
## Proposed Solution
### Option 1: Improve Probe Mechanism (Recommended)
- Add explicit test for image input capability during probing
- Use model API to test vision endpoint
- Parse Zhipu AI's model response correctly
### Option 2: Naming Convention Detection
- Auto-detect vision models by naming patterns:
- Models ending with `v` (e.g., `glm-4v`, `glm-4.6v`)
- Models containing `vision` or `flash`
- Add Zhipu AI-specific detection rules
### Option 3: Manual Configuration Override
- Allow users to manually specify `supports_multimodal: true`
- Add per-model configuration override option
- Provide documentation for vision model setup
### Option 4: Provider-Specific Profiles
- Add Zhipu AI-specific provider profile with vision model list
- Include known Zhipu vision models: `glm-4v`, `glm-4.6v-flash`, `glm-5v`
- Auto-apply correct settings for these models
## Additional Information
### Zhipu AI Vision Models
According to Zhipu AI documentation, these models support vision:
- **GLM-4V**: Base vision model
- **GLM-4V-Plus**: Enhanced vision model
- **GLM-4.6V-Flash**: Fast vision model (flash variant)
- **GLM-5V**: Latest vision model
### Naming Convention
Zhipu uses `v` suffix to indicate vision capability:
- `glm-4v` = GLM-4 with Vision
- `glm-4.6v-flash` = GLM-4.6 Vision Flash (fast variant)
### Alternative Names
The model may appear as:
- `glm-4.6v-flash` (with `v` in name)
- `glm-4v-flash` (alternate naming)
- `glm-flash-vision` (possible alias)
## Request for Guidance
1. **How should we configure multimodal models properly?**
- Is there a specific configuration field or option?
- Should we edit provider JSON directly?
- Any special setup steps?
2. **Is the probe mechanism known to have limitations?**
- Are there known issues with certain providers?
- How can we help improve detection accuracy?
3. **What's the recommended workaround for now?**
- Manual JSON editing?
- Specific provider configuration?
- Temporary solution?
## Checklist
- [x] Searched existing issues (no duplicates found)
- [x] Provided version information
- [x] Provided OS information
- [x] Detailed reproduction steps
- [x] Explained expected vs actual behavior
- [x] Included configuration evidence
- [x] Analyzed possible root causes
- [x] Proposed solution options
- [x] Requested specific guidance
## Priority
**Medium-High**: This affects legitimate use of vision models and requires manual intervention to fix. Impact is limited to Zhipu AI vision models but affects users who want to use these models' vision capabilities.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.