[Feature request] Support RAG

English Translation

Problem to solve:
When using arbitrary LLMs, they can usually plan concrete operations such as browser clicks reasonably well. But when the user gives a high-level instruction or one that is tightly tied to the product domain, the LLM may not understand what the user means or what the product concepts mean in the current context. This also happens with compound instructions, such as "search for a certain product". Even with UI-Tars, the LLM still lacks product knowledge.

In testing, I prefer allowing people in different roles to participate in AI-powered automation testing. That means the final automation is not just translating "click " into page.getLocator(x).click(), but handling more complex instructions that include many product-specific concepts.

Suggested implementation:

1. Provide basic RAG capability for Midscene.
2. This could be implemented in two phases:
   1. Add an extractedRAG input parameter to ai() or aiAction(). In the first phase, append this information to the end of the existing system message, for example under ## Additional Product Knowledge.
   2. Add a feature to retrieve RAG from external embeddings. When the user provides a query and a remote embeddings service (for example, embeddings deployed with Ollama and stored in local Redis or Postgres), Midscene can send the query to that embeddings service, retrieve the relevant context, and then feed it into step 2.1.
3. Midscene does not need to provide RAG indexing itself. Many enterprises already have their own RAG system, and Midscene only needs to reuse it.

Initial test:
The following report uses UITars-7B-SFT, L10S, 4 x GPU. RAG was manually added to the action prompt.

Next steps:

• I want to ask whether you agree that this is a valid requirement.
• If so, I can implement it soon.

Thank you.

想解决的问题：
使用任意LLM时，他们都可以还不错的plan一个操作 - 例如一个浏览器操作 - “点击 哪个按钮”， 而当用户给出的指令是一个high-level指令时或指令和产品非常相关（例如 - “我创建一个sticky note”， LLM不明白怎么创建，也不明白在当前的context下什么是sticky note ），又或多个联合指令 - （例如 - 我搜索一个XX产品）， LLM并不能很好的理解这个指令，即使使用UI-Tars， 因为LLM并不理解这个产品。

在测试中 - 我更倾向于让不同的角色的人参与到AI-enpowered的自动化测试中，因此最终的自动化测试就不是简单的让LLM 翻译“click - ” 到 “page.getLocator(x).click()”， 而是更为复杂的，且带有很多被测产品特色的测试指令。

建议的实现方法：
1 . 为midscene提供基本的RAG功能
2. 可以分为2步走
2.1 - 在ai() 或 aiAction() 时， 可以增加一个输入变量 extractedRAG， 第一步就是将这些信息附在原来system message 的最下面，例如## Additional Production Knowledge
2.2 - 新增一个从其他embeddings中获取rag的功能， 即当用户提供了一个query和远端embeddings时（例如我们将embeddings部署在ollama, 使用本地redis/postgres）， midscene 可以发送query 到远端的embeddings 去查询，然后在添加到2.1的步骤中

3. midscene将不提供RAG index的功能，因为对于很多企业来说， 他们已经建立了自己的RAG，我们只需要复用即可。

初步的测试：
以下测试报告使用 UITars-7B-SFT, L10S, 4 x GPU, 手动模拟增加了RAG到action的prompt。
Midscene report

下一步：

• 我想咨询你们是否认可这个需求
• 如何认可，我可以最近实现一下

谢谢

Start by locating the ai() and aiAction() entry points and how the existing system message is assembled. Scope the first phase around the proposed extractedRAG input and confirm that it is appended under Additional Product Knowledge; treat external embeddings retrieval as a separate follow-up phase.