coze-dev / coze-dev/coze-studio

Knowledge Base: add a standalone recall test for retrieval tuning

Open
#2,762 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
21.6k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

When tuning a knowledge base, users currently need to go through an Agent or Workflow debugging flow to inspect what was actually retrieved. That works for debugging a complete downstream request, but it is relatively heavy when the goal is only to answer a narrower question:

> For this knowledge base and this query, which chunks are retrieved under a given retrieval configuration, with what scores and sources?

There are several related community signals:

- In #2218, a user directly suggested adding a recall/retrieval test for evaluating a knowledge base.
- #2334 and #2459 report difficulty diagnosing poor chunking / retrieval quality.
- #1857 reports large differences between direct Milvus retrieval and Coze Workflow retrieval under apparently similar inputs.

The problem is not that Coze has no retrieval observability at all. Agent/Workflow debugging can already show recalled chunks. The missing piece is a lightweight **knowledge-base-level retrieval tuning loop** that does not require first constructing or entering a downstream orchestration flow.

**Describe the solution you'd like**

Add a standalone **Recall Test / Retrieval Playground** on the Knowledge Base detail page.

A minimal version could:

1. Let the user enter a single query.
2. Let the user provide temporary, test-scoped retrieval settings such as search strategy, TopK, and MinScore.
3. Reuse the same production knowledge retrieval path rather than introducing a separate retrieval implementation.
4. Show ranked retrieved chunks with at least final score and source document information.
5. Make zero-result cases explicit together with the active test settings.
6. Keep all test settings ephemeral: running a test should not modify the Knowledge Base itself or any Agent/Workflow configuration.

For a first version, I would intentionally keep the scope small: no batch dataset evaluation, no Recall@K / MRR / NDCG metrics, no experiment history, and no retrieval algorithm changes.

**Describe alternatives you've considered**

- **Use Agent/Workflow debugging only.** This is useful for end-to-end debugging, but it mixes knowledge retrieval diagnosis with downstream orchestration and makes quick retrieval tuning slower.
- **Improve retrieval quality directly.** That is still important, but repeated reports such as #1857 / #2334 are difficult to diagnose when users cannot isolate query → retrieval result → score → configuration at the Knowledge Base layer.
- **Expose more fields only in the Workflow knowledge-retrieval node.** #2253 is a useful historical example: maintainers noted that exposing more retrieval data requires coordinated frontend and backend behavior. A Knowledge-level test surface seems cleaner for this specific tuning workflow than only extending downstream node output.

**Additional context**

The current codebase already appears to have much of the underlying foundation:

- the knowledge retrieval domain path returns retrieved slices with scores;
- the Knowledge Base has its own IDE/detail surface where a test action could naturally live;
- the public frontend still contains an autogenerated `RetrieveTestReq/Resp` contract and a generated client path for `/api/devops/knowledge_platform/v1/retrieve_test` originating from the earlier Bytedance code mirror.

I am **not** assuming that the old generated `RetrieveTest` contract is intended to be restored in the open-source product; it may be an internal-only historical artifact. I am mentioning it only because it suggests that this workflow has been modeled before.

Before implementing, I would like to align with the maintainers on three points:

1. Is a standalone Knowledge-level retrieval test something you want in the open-source product, or should retrieval inspection intentionally remain inside Agent/Workflow debugging?
2. Is the existing generated `RetrieveTest` contract still conceptually relevant, or should an OSS-native API be designed around the current knowledge retrieval domain path?
3. If the direction is welcome, which Knowledge UI surface and API boundary would you prefer for a minimal first version?

If this direction fits the project, I would be happy to take ownership of the scoped implementation after we align on the boundary.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing knowledge retrieval domain path and the Knowledge Base IDE/detail surface. Review the generated RetrieveTestReq/Resp contract and client path for /api/devops/knowledge_platform/v1/retrieve_test, without assuming it should be restored. Done means maintainers agree on the API and UI boundary, then users can run an ephemeral query test and see ranked chunks, scores, sources, settings, and explicit zero-result cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, backend-api-design, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.