abetlen / abetlen/llama-cpp-python
[Bug] DFlash 2 speculative decoding crashes and fails in Python due to missing ctx_other and non-autoregressive graph execution
- Ngôn ngữ chính
- Python
- Star
- 10.6k
- Fork
- 1.4k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
# Prerequisites
- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the README.md.
- [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the Discussions, and have a new bug or useful enhancement to share.
# Expected Behavior
When loading the target model `utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF` (`Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf`) together with the DFlash 2 draft model `incoai/Qwen3.8-27B-DFlash2-GGUF` (`Qwen3.8-27B-DFlash2-Q4_K_M.gguf`), `llama-cpp-python` should link the draft context to the target context via `cparams.ctx_other = target_context` and execute the native C++ block-diffusion speculative decoding pipeline, producing speculative speedups.
# Current Behavior
1. Calling `draft_llm = Llama(model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf")` fails in GGML with:
`dflash requires ctx_other to be set` -> `ValueError: Failed to create llama_context`
because DFlash 2 sidecar GGUF files do not contain their own `lm_head` / `output.weight` tensors and require `ctx_other` to be set during context initialization.
2. Setting `draft_llm.model = draft_model_ptr` fails with:
`AttributeError: property 'model' of 'Llama' object has no setter`.
3. Calling `LlamaDraftModel(draft_llm, num_pred_tokens=5)` fails with:
`TypeError: LlamaDraftModel() takes no arguments` because `LlamaDraftModel` is an abstract base class.
4. When subclassing `LlamaDraftModel` and returning a Python `list`, it crashes inside `llama.py` with:
`AttributeError: 'list' object has no attribute 'astype'`.
5. When returning a `numpy.ndarray` with `dtype=np.intc`, the Python loop calls `draft_llm.sample()`. This bypasses the C++ DFlash 2 pipeline (target hidden layer extraction `llama_get_embeddings_layer_inp`, encoder pass `llama_encode`, and candidate lattice selection `build_post_sampling`), resulting in a 0% draft acceptance rate and dropping generation speed to baseline autoregressive throughput (~34.27 tok/s).
# Environment and Context
* **Hardware:** NVIDIA RTX PRO 6000 Blackwell Server Edition (`sm_120`), 24+ GB VRAM
* **Environment:** Hugging Face ZeroGPU Space
* **Operating System:** Debian GNU/Linux 13 (trixie) / Linux 6.12.94-123.192.amzn2023.x86_64
* **glibc Version:** Debian GLIBC 2.41-12
* **Python Version:** 3.12.12
* **NVIDIA Driver:** 580.159.03 (CUDA 13.0)
* **llama-cpp-python:** v0.3.35 compiled against submodule `vendor/llama.cpp` using:
* **Repository:** https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4
* **Branch:** `dflash2`
* **Commit Hash:** `5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4`
* **Upstream Pull Request:** https://github.com/ggml-org/llama.cpp/pull/27342/changes
# Steps to Reproduce
1. Build `llama-cpp-python` with `vendor/llama.cpp` checked out at `z-lab/llama.cpp-fork` commit `5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4`.
2. Download target model `utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF` and draft model `incoai/Qwen3.8-27B-DFlash2-GGUF`.
3. Attempt to initialize the draft model in Python:
```python
import llama_cpp
from llama_cpp import Llama
# 1. Target model initialization succeeds:
target_llm = Llama(
model_path="Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf",
n_gpu_layers=-1,
n_ctx=8192,
)
# 2. Draft model initialization fails here:
draft_llm = Llama(
model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf",
n_gpu_layers=-1,
n_ctx=8192,
)
```
# Failure Logs
**Context creation failure:**
```text
File "app.py", line 58, in get_or_load_model
draft_llm = Llama(model_path=DRAFT_MODEL_PATH, ...)
File ".../llama_cpp/llama.py", line 415, in __init__
internals.LlamaContext(
File ".../llama_cpp/_internals.py", line 266, in __init__
raise ValueError("Failed to create llama_context")
ValueError: Failed to create llama_context
```
**Attribute setter failure:**
```text
File "app.py", line 94, in get_or_load_dflash2_model
draft_llm.model = draft_model_ptr
AttributeError: property 'model' of 'Llama' object has no setter
```
**LlamaDraftModel constructor failure:**
```text
File "app.py", line 108, in get_or_load_dflash2_model
target_llm.draft_model = LlamaDraftModel(draft_model=draft_llm, num_pred_tokens=5)
TypeError: LlamaDraftModel() takes no arguments
```
**List vs NumPy array failure:**
```text
File ".../llama_cpp/llama.py", line 1022, in generate
draft_tokens.astype(int)[
AttributeError: 'list' object has no attribute 'astype'
```
---
# Additional Notes & Disclaimers
* **Goal:** I am trying to run **DFlash 2** (`Qwen3.8-27B-DFlash2-Q4_K_M.gguf`) on a **Hugging Face ZeroGPU Space** with `llama-cpp-python`.
* **Specific Fork:** The C++ code is from https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 (PR https://github.com/ggml-org/llama.cpp/pull/27342/changes).
* **Disclaimer on other speculative methods:** The mention of EAGLE was suggested by an AI assistant during our discussion; I cannot personally confirm EAGLE's implementation details. I also do not know with certainty about built-in MTP, DFlash v1, or DSpark in `llama-cpp-python`, though DFlash v1 and DSpark may already be present in upstream `llama.cpp`.
* **AI Disclosure:** This bug report was formatted with the help of an AI assistant at my request during a live debugging session. All steps, code snippets, stack traces, and reproduction logs come directly from my testing on Hugging Face ZeroGPU.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.