antirez / antirez/ds4

Model reaches correct answer internally but won't commit — repeated web_search loops on small reasoning puzzles

Aperta
#646 3 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
22.3k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

## Setup

- Build: `54b36ed` (main)
- Model: `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf` (0731 weight release, IQ2XXS/w2Q2K routed experts)
- Runtime: `ds4-agent` via `./agent.sh`, ctx 131072, Metal backend on an Apple Silicon Mac (M5 Max)
- No DSpark/MTP in use

## Symptom

On small logic-puzzle prompts, the agent burns a large amount of reasoning (~30k tokens) and, based on the reasoning trace, appears to arrive at the correct answer internally — but instead of committing to a final response it repeatedly invokes `web_search_call` to try to verify the answer, and its confidence in its own derived answer never stabilizes.

Two test cases:

1. **Numeric transform puzzle:**
```
11118888888855 -> 118885 | 79999775555 -> 99755 | AAABBBYUDD -> ?
```
Here the run eventually stopped on its own — it found an answer via web search and checked its own derived answer against that, rather than trusting its internal reasoning.

2. **Lateral-thinking riddle:**
```
A feather → Heavy
A ton of bricks → Light
A gold bar → Heavy
A secret → ?
```
This one never stopped by itself — had to be manually interrupted. The model appeared to keep flip-flopping between candidate answers, unable to settle on which one was correct.

I quantized Qwen3.6-35B independently a while back and saw a similar pattern there too (excessive deliberation without commitment on small puzzle-style prompts), so my guess is this points at a routing failure in the MoE experts rather than something specific to this model or quant — possibly aggravated by the low-bit (IQ2XXS/w2Q2K) routed-expert quantization here. Not sure if this is fixable or just an inherent cost of this quant tier, but wanted to flag it in case it's useful data.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.