abetlen / abetlen/llama-cpp-python

[Bug] DFlash 2 speculative decoding crashes and fails in Python due to missing ctx_other and non-autoregressive graph execution

Aberta
#2,361 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
10.6k
Forks
1.4k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

# Prerequisites

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the README.md.
- [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the Discussions, and have a new bug or useful enhancement to share.

# Expected Behavior

When loading the target model `utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF` (`Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf`) together with the DFlash 2 draft model `incoai/Qwen3.8-27B-DFlash2-GGUF` (`Qwen3.8-27B-DFlash2-Q4_K_M.gguf`), `llama-cpp-python` should link the draft context to the target context via `cparams.ctx_other = target_context` and execute the native C++ block-diffusion speculative decoding pipeline, producing speculative speedups.

# Current Behavior

1. Calling `draft_llm = Llama(model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf")` fails in GGML with:
`dflash requires ctx_other to be set` -> `ValueError: Failed to create llama_context`
because DFlash 2 sidecar GGUF files do not contain their own `lm_head` / `output.weight` tensors and require `ctx_other` to be set during context initialization.
2. Setting `draft_llm.model = draft_model_ptr` fails with:
`AttributeError: property 'model' of 'Llama' object has no setter`.
3. Calling `LlamaDraftModel(draft_llm, num_pred_tokens=5)` fails with:
`TypeError: LlamaDraftModel() takes no arguments` because `LlamaDraftModel` is an abstract base class.
4. When subclassing `LlamaDraftModel` and returning a Python `list`, it crashes inside `llama.py` with:
`AttributeError: 'list' object has no attribute 'astype'`.
5. When returning a `numpy.ndarray` with `dtype=np.intc`, the Python loop calls `draft_llm.sample()`. This bypasses the C++ DFlash 2 pipeline (target hidden layer extraction `llama_get_embeddings_layer_inp`, encoder pass `llama_encode`, and candidate lattice selection `build_post_sampling`), resulting in a 0% draft acceptance rate and dropping generation speed to baseline autoregressive throughput (~34.27 tok/s).

# Environment and Context

* **Hardware:** NVIDIA RTX PRO 6000 Blackwell Server Edition (`sm_120`), 24+ GB VRAM
* **Environment:** Hugging Face ZeroGPU Space
* **Operating System:** Debian GNU/Linux 13 (trixie) / Linux 6.12.94-123.192.amzn2023.x86_64
* **glibc Version:** Debian GLIBC 2.41-12
* **Python Version:** 3.12.12
* **NVIDIA Driver:** 580.159.03 (CUDA 13.0)
* **llama-cpp-python:** v0.3.35 compiled against submodule `vendor/llama.cpp` using:
* **Repository:** https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4
* **Branch:** `dflash2`
* **Commit Hash:** `5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4`
* **Upstream Pull Request:** https://github.com/ggml-org/llama.cpp/pull/27342/changes

# Steps to Reproduce

1. Build `llama-cpp-python` with `vendor/llama.cpp` checked out at `z-lab/llama.cpp-fork` commit `5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4`.
2. Download target model `utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF` and draft model `incoai/Qwen3.8-27B-DFlash2-GGUF`.
3. Attempt to initialize the draft model in Python:

```python
import llama_cpp
from llama_cpp import Llama

# 1. Target model initialization succeeds:
target_llm = Llama(
model_path="Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf",
n_gpu_layers=-1,
n_ctx=8192,
)

# 2. Draft model initialization fails here:
draft_llm = Llama(
model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf",
n_gpu_layers=-1,
n_ctx=8192,
)
```

# Failure Logs

**Context creation failure:**
```text
File "app.py", line 58, in get_or_load_model
draft_llm = Llama(model_path=DRAFT_MODEL_PATH, ...)
File ".../llama_cpp/llama.py", line 415, in __init__
internals.LlamaContext(
File ".../llama_cpp/_internals.py", line 266, in __init__
raise ValueError("Failed to create llama_context")
ValueError: Failed to create llama_context
```

**Attribute setter failure:**
```text
File "app.py", line 94, in get_or_load_dflash2_model
draft_llm.model = draft_model_ptr
AttributeError: property 'model' of 'Llama' object has no setter
```

**LlamaDraftModel constructor failure:**
```text
File "app.py", line 108, in get_or_load_dflash2_model
target_llm.draft_model = LlamaDraftModel(draft_model=draft_llm, num_pred_tokens=5)
TypeError: LlamaDraftModel() takes no arguments
```

**List vs NumPy array failure:**
```text
File ".../llama_cpp/llama.py", line 1022, in generate
draft_tokens.astype(int)[
AttributeError: 'list' object has no attribute 'astype'
```

---

# Additional Notes & Disclaimers

* **Goal:** I am trying to run **DFlash 2** (`Qwen3.8-27B-DFlash2-Q4_K_M.gguf`) on a **Hugging Face ZeroGPU Space** with `llama-cpp-python`.
* **Specific Fork:** The C++ code is from https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 (PR https://github.com/ggml-org/llama.cpp/pull/27342/changes).
* **Disclaimer on other speculative methods:** The mention of EAGLE was suggested by an AI assistant during our discussion; I cannot personally confirm EAGLE's implementation details. I also do not know with certainty about built-in MTP, DFlash v1, or DSpark in `llama-cpp-python`, though DFlash v1 and DSpark may already be present in upstream `llama.cpp`.
* **AI Disclosure:** This bug report was formatted with the help of an AI assistant at my request during a live debugging session. All steps, code snippets, stack traces, and reproduction logs come directly from my testing on Hugging Face ZeroGPU.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.