nextcloud / nextcloud/context_chat_backend

Cross-language document search breaks because E5 queries are sent without instructions

Open Beginner friendly
#331 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug good first issue
Dominant language
Python
Stars
27
Forks
24
Avg merge
3d 4h
Merged PRs (30d)
6

Description

Which version of Nextcloud are you using?

34.0.2

Which version of PHP context_chat are you using?

5.4.0

Which version of backend context_chat are you using?

5.4.1

Nextcloud deployment method?

Docker Compose via Coolify

Describe the Bug

Hi! We ran into a problem with multilingual document search using the default multilingual-e5-large-instruct embedding model.

The backend’s default configuration selects that model here:
https://github.com/nextcloud/context_chat_backend/blob/v5.4.1/config.cpu.yaml#L21-L43

The model documentation says that queries must use this format:

Instruct: <task description>
Query: <query>

Model documentation: https://huggingface.co/intfloat/multilingual-e5-large-instruct
To quote,

  1. Do I need to add instructions to the query?
    Yes, this is how the model is trained, otherwise you will see a performance degradation.

However, NetworkEmbeddings.embed_query() currently passes the original query straight to the embedding endpoint:

https://github.com/nextcloud/context_chat_backend/blob/v5.4.1/context_chat_backend/network_em.py#L178-L179

def embed_query(self, text: str) -> list[float]:
    return self._get_embedding(text)

This seems to make cross-language retrieval unreliable, despite the bundled model being multilingual. We tested, and when following the instruct/query format, we get much better retrieval.

To Reproduce
  1. Create and index a German md document containing a unique term, for example:
Ein Wasserbrandango ist ein australisches Tier.
Man nennt es auch FWFABZSFBUASJJNSF.
  1. Wait until the document has been indexed.
  2. Ask in English: What is a FWFABZSFBUASJJNSF?
  3. Context Chat does not retrieve the German document and answers that it does not know, or returns unrelated sources.
  4. Send the same unchanged query to the embedding model using the instruction format:
Instruct: Given a user question, retrieve relevant passages that answer the question, even when they are written in another language.
Query: What is a FWFABZSFBUASJJNSF?
  1. The correct German document is retrieved.

Expected result
The default bundled embedding model should receive queries in its documented instruction format, so a question in one language can retrieve relevant documents written in another language.

PHP logs (Warning these might contain sensitive information)

No response

Ex-App logs (Warning these might contain sensitive information)

No response

Server logs (if applicable)

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with config.cpu.yaml to confirm the default multilingual-e5-large-instruct model, then inspect NetworkEmbeddings.embed_query() in context_chat_backend/network_em.py. Reproduce the issue with the supplied German document and English query, and verify that the documented Instruct/Query format enables retrieval of the cross-language result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning, search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.