zai-org / zai-org/feedback

[Feature Request] ZCode: Add GLM-5.3V Vision Model & Improve Vision Capabilities (Enable by Default)

Open
#310 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority: P2
Dominant language
No language data
Stars
22
Forks
1
PR merge metrics
No merged PRs in 30d

Description

THIS FEATURE REQUEST SUBMITTED ON BEHALF OF A DISCORD USER: Judikael Nedev

Feature Request: ZCode Vision Model Enhancements — GLM-5.3V + Default Vision Support

📋 Summary

Two related vision capability improvements for ZCode:

  1. Add GLM-5.3V — Request for a vision-capable variant of the GLM-5.3 model series (currently only GLM-5-Turbo has a vision variant: GLM-5V-Turbo)
  2. Enable Vision Models by Default — Currently users must manually edit model names to add "V" suffix; vision models should be readily available without configuration hacks

🎯 Problem Description

Current State
Model Available Vision Capable
GLM-5.3 ✅ Yes ❌ No (no V variant)
GLM-5.2 ✅ Yes ❌ No (no V variant)
GLM-5-Turbo ✅ Yes ⚠️ Only if renamed to GLM-5V-Turbo
GLM-5V-Turbo 🔧 Hidden ✅ Yes (requires manual edit)
User Pain Points
Pain Point 1: No Vision Variant for Flagship Model
User wants: GLM-5.3 with vision capabilities (GLM-5.3V)
Reality:   Does not exist — must use older/smaller GLM-5V-Turbo instead
Impact:    Cannot use best reasoning model + vision together

Use case from reporter:

"For independent loop with self testing, the Image MCP is not sufficient. When I'm starting my agent on task autonomously, I would like to make him verify with a screenshot tool his work."

Judikael Nedev [TKN]

Scenario: Agent-driven development where AI needs to:

  • Take screenshots of its work
  • Verify UI/rendering visually
  • Self-test output using visual inspection
  • Image MCP tools alone are insufficient for this workflow
Pain Point 2: Manual Configuration Required
Current Workflow (painful):
1. Open ZCode settings
2. Find model configuration
3. Manually rename: GLM-5-Turbo → GLM-5V-Turbo
4. Hope it works (undocumented feature)
5. If config resets, do it all again

Desired Workflow:
1. Select "GLM-5V-Turbo" from model dropdown ← Should exist!
2. Or toggle "Vision Mode" on any supported model
3. Start working with images/screenshots immediately
User Feedback

"A vision encoder variant would be great. And in ZCode I have to manually edit the GLM-5-Turbo, add the V to have a model capable of seeing my pictures."

"Yes, but I was wondering why it's not enabled by default in ZCode."

Judikael Nedev [TKN] (Discord)


✅ Proposed Solutions

Solution 1: Add GLM-5.3V Model

Request: Create and deploy a vision-capable variant of GLM-5.3.

Expected Specs (based on GLM-5.3):

{
  "GLM-5.3V": {
    "name": "glm-5.3-vision",
    "reasoning": {
      "enabled": true,
      "variants": ["high", "low", "max"],
      "defaultVariant": "max"
    },
    "limit": {
      "context": 1000000,
      "output": 128000,
      "maxImageResolution": "4096x4096",
      "maxImagesPerRequest": 16
    },
    "modalities": {
      "input": ["text", "image"],
      "output": ["text"]
    },
    "visionFeatures": {
      "screenshotAnalysis": true,
      "uiUnderstanding": true,
      "chartReading": true,
      "codeScreenshotReview": true,
      "documentScanning": true
    }
  }
}

Use Cases Enabled:

Use Case Description
Agent Self-Testing AI takes screenshots of its work, verifies visually
UI Development AI can see what it's building, catch visual bugs
Document Analysis Read screenshots of docs, diagrams, whiteboards
Code Review Screenshot code editor, review formatting/layout
Visual Debugging See error messages, browser console, devtools

Solution 2: Enable Vision Models by Default in ZCode
Option A: Expose Vision Models in Dropdown
┌─────────────────────────────────────────────┐
│  Select Model:                             │
│  ┌───────────────────────────────────────┐ │
│  │ 🧠 GLM-5.3                    (Text)  │ │
│  │ 👁️ GLM-5.3V              (Text+Image)│ │  ← NEW
│  │ 🧠 GLM-5.2                    (Text)  │ │
│  │ 👁️ GLM-5.2V              (Text+Image)│ │  ← NEW
│  │ 🚀 GLM-5-Turbo                (Text)  │ │
│  │ 👁️ GLM-5V-Turbo        (Text+Image)│ │  ← NOW VISIBLE
│  └───────────────────────────────────────┘ │
│                                             │
│  💡 Models with 👁️ support image input     │
└─────────────────────────────────────────────┘```

#### Option B: Vision Toggle on Supported Models

┌─────────────────────────────────────────────┐
│ Model: [GLM-5.3 ▼] │n│ │
│ ☑️ Enable Vision Mode │
│ └→ Switches to GLM-5.3V automatically ││ ││ Image Settings: │n│ • Max resolution: 4096 × 4096 │
│ • Max images per message: 16 ││ • Auto-compress large images │└─────────────────────────────────────────────┘```

Option C: Auto-Detect When Images Present
// Intelligent model switching
function selectModelForInput(userMessage, attachments) {
  const hasImages = attachments.some(a => a.type.startsWith('image/'));
  const baseModel = userSettings.preferredModel; // e.g., "GLM-5.3"
  
  if (hasImages) {
    const visionVariant = baseModel + 'V'; // e.g., "GLM-5.3V"
    if (isModelAvailable(visionVariant)) {
      return visionVariant;
    }
    // Fallback to available vision model
    return getBestAvailableVisionModel();
  }
  
  return baseModel;
}

Recommended: Implement Option A first (simplest), add Option C later (best UX).


🔧 Technical Implementation

ZCode Config Changes

Current (user must hack):

// ~/.zcode/v2/config.json
{
  "models": {
    "GLM-5-Turbo": { ... }  // User changes to GLM-5V-Turbo manually
  }
}

Proposed (vision models included out-of-box):

// ~/.zcode/v2/config.json (after fix)
{
  "models": {
    "GLM-5.3": {
      "reasoning": { "enabled": true, "variants": ["high", "low", "max"] },
      "limit": { "context": 1000000, "output": 128000 },
      "modalities": { "input": ["text"], "output": ["text"] }
    },
    "GLM-5.3V": {                                    // ← NEW
      "reasoning": { "enabled": true, "variants": ["high", "low", "max"] },
      "limit": { "context": 1000000, "output": 128000 },
      "modalities": { "input": ["text", "image"], "output": ["text"] },
      "vision": { "maxResolution": "4096x4096", "maxImages": 16 }
    },
    "GLM-5-Turbo": { ... },
    "GLM-5V-Turbo": {                                  // ← Now visible
      "baseModel": "GLM-5-Turbo",
      "modalities": { "input": ["text", "image"], "output": ["text"] }
    }
  }
}
API Endpoint Support
// Vision-enabled chat completion
POST /api/coding/v4/chat/completions

{
  "model": "GLM-5.3V",           // or "GLM-5V-Turbo"
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "What's wrong with this screenshot?" },
        { 
          "type": "image_url", 
          "image_url": { 
            "url": "data:image/png;base64,...",
            "detail": "high" 
          } 
        }
      ]
    }
  ]
}

🎯 Vision Use Cases for Agent Workflows

Scenario: Autonomous Agent with Visual Verification
┌─────────────────────────────────────────────────────────────┐│                  AGENT SELF-TESTING LOOP                      │├─────────────────────────────────────────────────────────────┤│                                                             ││  1. Agent completes coding task                              ││       ↓                                                      ││  2. Agent takes screenshot of running application            ││       ↓                                                      ││  3. Agent sends screenshot to GLM-5.3V for analysis          ││       ↓                                                      ││  4. GLM-5.3V responds:                                       ││     ✓ "UI looks correct, button positioned properly"        ││     ✗ "Color scheme doesn't match design spec here..."      ││     ✗ "This error message appeared unexpectedly..."         ││       ↓                                                      ││  5. Agent fixes issues based on visual feedback              ││       ↓                                                      ││  6. Repeat until GLM-5.3V approves                           ││                                                             ││  Result: Higher quality autonomous output!                   ││                                                             │└─────────────────────────────────────────────────────────────┘```

**Why Image MCP Alone Isn't Enough:**

| Limitation | Vision Model Advantage |
|-------------|------------------------|
| MCP returns structured data only | Model sees full visual context |
| Can't assess layout/aesthetics | Understands spatial relationships |
| Can't notice unexpected visual bugs | Catches things you didn't ask to check |
| Requires pre-defined inspection steps | Free-form visual reasoning |

---

## 📊 Success Metrics

| Metric | Target |
|--------|--------|
| Vision model adoption | >25% of users try within 1 month |
| Agent task success rate | +15% improvement for visual tasks |
| Manual config edits | -90% reduction (users don't need to hack config) |
| Support tickets | Reduction in "how do I enable vision?" requests |

---

## 🔄 Implementation Priority

| Phase | Feature | Effort | Value |
|-------|---------|--------|-------|
| **P0** | Expose GLM-5V-Turbo in ZCode model list | Low | High |
| **P0** | Remove need for manual "V" edit | Low | High |
| **P1** | Develop & deploy GLM-5.3V | High | Very High |
| **P1** | Add vision toggle UI in settings | Medium | Medium |
| **P2** | Auto-detect images → switch to vision model | Medium | Medium |
| **P2** | GLM-5.2V variant | Medium | Low |
| **P3** | Advanced vision features (video, multi-image comparison) | High | Low |

**Recommended MVP:** Ship P0 immediately (config fix), prioritize P1 (GLM-5.3V development).

---

## 🧪 Testing Scenarios

1. **Basic Vision:** Select GLM-5V-Turbo from dropdown → send screenshot → get description
2. **New Model:** Select GLM-5.3V → send complex UI screenshot → detailed analysis
3. **Agent Loop:** Run autonomous task with self-test screenshots → verify quality improvement
4. **Fallback:** Send image to non-vision model → graceful error or auto-switch prompt
5. **Config Persistence:** Restart ZCode after selecting vision model → still selected
6. **Performance:** Compare token usage: text-only vs vision prompts on same model

---

## 💡 Additional Context

- **Affected Product:** ZCode IDE
- **Primary Use Case:** Autonomous agent workflows with visual self-verification
- **Reporter:** Judikael Nedev [TKN] (Discord)
- **Discord Thread:** https://discord.com/channels/1346756824233148527/1539266816705167502
- **Related Issues:** Image MCP limitations for agent self-testing
- **Competitor Reference:** Claude has native vision across all model tiers; GPT-4o/GPT-4o-mini both vision-native

---
*Submitted by:*  
Regards,
**Roman** (Discord: bignavi_x)  
- Z.ai Volunteer Ambassador 
- Tech resources channel: https://t.me/VibeCodePrompterSystem
- Tech Blog: http://claw.rommark.dev/blog

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the ~/.zcode/v2/config.json examples and the POST /api/coding/v4/chat/completions payload described in the issue. Separate the requested MVP of exposing GLM-5V-Turbo from the larger GLM-5.3V, toggle, and auto-detection proposals. Done should include selectable vision models, persistent configuration, and the listed basic vision, fallback, and restart scenarios.

Written by the indexing model from the issue text.

Assessment

Domain
ai, api, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.