antirez / antirez/ds4

Model reaches correct answer internally but won't commit — repeated web_search loops on small reasoning puzzles

Ouverte
#646 3 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
22.3k
Forks
2.1k
Merge moyen
1 j 3 h
PR mergées (30 j)
4

Description

## Setup

- Build: `54b36ed` (main)
- Model: `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf` (0731 weight release, IQ2XXS/w2Q2K routed experts)
- Runtime: `ds4-agent` via `./agent.sh`, ctx 131072, Metal backend on an Apple Silicon Mac (M5 Max)
- No DSpark/MTP in use

## Symptom

On small logic-puzzle prompts, the agent burns a large amount of reasoning (~30k tokens) and, based on the reasoning trace, appears to arrive at the correct answer internally — but instead of committing to a final response it repeatedly invokes `web_search_call` to try to verify the answer, and its confidence in its own derived answer never stabilizes.

Two test cases:

1. **Numeric transform puzzle:**
```
11118888888855 -> 118885 | 79999775555 -> 99755 | AAABBBYUDD -> ?
```
Here the run eventually stopped on its own — it found an answer via web search and checked its own derived answer against that, rather than trusting its internal reasoning.

2. **Lateral-thinking riddle:**
```
A feather → Heavy
A ton of bricks → Light
A gold bar → Heavy
A secret → ?
```
This one never stopped by itself — had to be manually interrupted. The model appeared to keep flip-flopping between candidate answers, unable to settle on which one was correct.

I quantized Qwen3.6-35B independently a while back and saw a similar pattern there too (excessive deliberation without commitment on small puzzle-style prompts), so my guess is this points at a routing failure in the MoE experts rather than something specific to this model or quant — possibly aggravated by the low-bit (IQ2XXS/w2Q2K) routed-expert quantization here. Not sure if this is fixable or just an inherent cost of this quant tier, but wanted to flag it in case it's useful data.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.