NVIDIA / NVIDIA/GenerativeAIExamples

Chain server fail to pull the result

Open
#269 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.2k
Forks
1.1k
Avg merge
10h 15m
Merged PRs (30d)
1

Description

2025/01/29 12:35:47.094 +00:00] [WARN] [retry/retry.go:104] ["retry func failed, reach max retry"] [traceID=5ce0d7f04403e4b765eefacedfbf5a5b] [attempt=2]
[2025/01/29 12:35:47.094 +00:00] [WARN] [proxy/task_search.go:581] ["search execute failed"] [traceID=5ce0d7f04403e4b765eefacedfbf5a5b] [nq=1] [error="failed to search/query delegator 2 for channel by-dev-rootcoord-dml_1_454401150931015494v0: fail to search on QueryNode 2: worker(2) query failed: Operator::GetOutput failed for [Operator:, plan node id: 11562] : => failed to search: config={{"ef":10,"k":40,"metric_type":"L2","span_id":"11dd9c06b163fd13","trace_flags":0,"trace_id":"5ce0d7f04403e4b765eefacedfbf5a5b"}} out of range in json: ef(10) should be larger than k(40) at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:411"] [errorVerbose="failed to search/query delegator 2 for channel by-dev-rootcoord-dml_1_454401150931015494v0: fail to search on QueryNode 2: worker(2) query failed: Operator::GetOutput failed for [Operator:, plan node id: 11562] : => failed to search: config={{"ef":10,"k":40,"metric_type":"L2","span_id":"11dd9c06b163fd13","trace_flags":0,"trace_id":"5ce0d7f04403e4b765eefacedfbf5a5b"}} out of range in json: ef(10) should be larger than k(40) at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:411\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:218\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 2 for channel by-dev-rootcoord-dml_1_454401150931015494v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:791\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:210\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:179\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:246\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (4) fail to search on QueryNode 2\nWraps: (5) worker(2) query failed: Operator::GetOutput failed for [Operator:, plan node id: 11562] : => failed to search: config={{"ef":10,"k":40,"metric_type":"L2","span_id":"11dd9c06b163fd13","trace_flags":0,"trace_id":"5ce0d7f04403e4b765eefacedfbf5a5b"}} out of range in json: ef(10) should be larger than k(40) at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:411\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the reported failure from internal/proxy/task_search.go:581 through internal/proxy/lb_policy.go and pkg/util/retry/retry.go, then inspect the VectorMemIndex.cpp:411 error showing ef(10) below k(40). Reproduce the search in the chain-server workflow and determine the expected result; done means the query returns its result without the reported retry failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.