docling-project / docling-project/docling

mistralai/Mixtral-8x7B-Instruct-v0.1 fails in Docling RAG example with HuggingFaceEndpoint: Task mismatch (text-generation vs conversational)

Open
#2,080 2 comments 0 reactions 0 assignees View on GitHub
bug triage/close-fixed
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug

Following the Docling documentation example for RAG pipelines, replacing the LLM with` mistralai/Mixtral-8x7B-Instruct-v0.1` results in:

`ValueError: Model mistralai/Mixtral-8x7B-Instruct-v0.1 is not supported for task text-generation and provider together. Supported task: conversational.`

The documentation shows usage with `task="text-generation"`, but this model is only supported for `conversational`.
Even when changing `task="conversational"`, the same error occurs, which makes it unclear how to integrate conversational-only models with the example workflow.

### Steps to reproduce

```
from pathlib import Path
from tempfile import mkdtemp
from dotenv import load_dotenv
import os
from langchain_core.prompts import PromptTemplate
from langchain_docling.loader import ExportType
from docling.chunking import HybridChunker
from langchain_docling import DoclingLoader
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_huggingface import HuggingFaceEndpoint

def _get_env_from_colab_or_os(key):
try:
from google.colab import userdata
try:
return userdata.get(key)
except userdata.SecretNotFoundError:
pass
except ImportError:
pass
return os.getenv(key)

load_dotenv()

HF_TOKEN = _get_env_from_colab_or_os("HF_TOKEN")
FILE_PATH = ["https://arxiv.org/pdf/2408.09869"]
EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"
GEN_MODEL_ID = "mistralai/Mixtral-8x7B-Instruct-v0.1"
EXPORT_TYPE = ExportType.DOC_CHUNKS
QUESTION = "Which are the main AI models in Docling?"
PROMPT = PromptTemplate.from_template(
"Context information is below.\n---------------------\n{context}\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: {input}\nAnswer:\n",
)
TOP_K = 3
MILVUS_URI = str(Path(mkdtemp()) / "docling.db")

loader = DoclingLoader(
file_path=FILE_PATH,
export_type=EXPORT_TYPE,
chunker=HybridChunker(tokenizer=EMBED_MODEL_ID),
)

docs = loader.load()
retriever = vectorstore.as_retriever(search_kwargs={"k": TOP_K})

llm = HuggingFaceEndpoint(
repo_id=GEN_MODEL_ID,
huggingfacehub_api_token=HF_TOKEN,
task="text-generation" # Also tried task="conversational"
)

question_answer_chain = create_stuff_documents_chain(llm, PROMPT)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

resp_dict = rag_chain.invoke({"input": QUESTION})
print(resp_dict["answer"])
```
Error:
`ValueError: Model mistralai/Mixtral-8x7B-Instruct-v0.1 is not supported for task text-generation and provider together. Supported task: conversational.`

What I Tried:

- Changing **task="text-generation"** → **task="conversational"**.

- Confirmed that **Mixtral-8x7B-Instruct-v0.1** works in HuggingFace directly with conversational/chat APIs.

- Using **OpenAI** model (**gpt-4o-mini**) with the same Docling example works fine.

- The error still occurs when switching to **task="conversational"** in Docling's **HuggingFaceEndpoint.**

### Docling version

**v2.44.0**

### Python version

**Python: 3.11**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.