nextcloud / nextcloud/context_chat_backend
Cross-language document search breaks because E5 queries are sent without instructions
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 27
- Forks
- 24
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 6
Description
Which version of Nextcloud are you using?
34.0.2
Which version of PHP context_chat are you using?
5.4.0
Which version of backend context_chat are you using?
5.4.1
Nextcloud deployment method?
Docker Compose via Coolify
Describe the Bug
Hi! We ran into a problem with multilingual document search using the default multilingual-e5-large-instruct embedding model.
The backend’s default configuration selects that model here:
https://github.com/nextcloud/context_chat_backend/blob/v5.4.1/config.cpu.yaml#L21-L43
The model documentation says that queries must use this format:
Instruct: <task description>
Query: <query>
Model documentation: https://huggingface.co/intfloat/multilingual-e5-large-instruct
To quote,
- Do I need to add instructions to the query?
Yes, this is how the model is trained, otherwise you will see a performance degradation.
However, NetworkEmbeddings.embed_query() currently passes the original query straight to the embedding endpoint:
def embed_query(self, text: str) -> list[float]:
return self._get_embedding(text)
This seems to make cross-language retrieval unreliable, despite the bundled model being multilingual. We tested, and when following the instruct/query format, we get much better retrieval.
To Reproduce
- Create and index a German md document containing a unique term, for example:
Ein Wasserbrandango ist ein australisches Tier.
Man nennt es auch FWFABZSFBUASJJNSF.
- Wait until the document has been indexed.
- Ask in English:
What is a FWFABZSFBUASJJNSF? - Context Chat does not retrieve the German document and answers that it does not know, or returns unrelated sources.
- Send the same unchanged query to the embedding model using the instruction format:
Instruct: Given a user question, retrieve relevant passages that answer the question, even when they are written in another language.
Query: What is a FWFABZSFBUASJJNSF?
- The correct German document is retrieved.
Expected result
The default bundled embedding model should receive queries in its documented instruction format, so a question in one language can retrieve relevant documents written in another language.
PHP logs (Warning these might contain sensitive information)
No response
Ex-App logs (Warning these might contain sensitive information)
No response
Server logs (if applicable)
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with config.cpu.yaml to confirm the default multilingual-e5-large-instruct model, then inspect NetworkEmbeddings.embed_query() in context_chat_backend/network_em.py. Reproduce the issue with the supplied German document and English query, and verify that the documented Instruct/Query format enables retrieval of the cross-language result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning, search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100