dgraph-io / dgraph-io/dgraph

v25.3.1, alpha errors "Please retry again, server is not ready to accept requests"

Open
#9,695 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
21.8k
Forks
1.6k
Avg merge
2d 5h
Merged PRs (30d)
9

Description

Describe the bug

we deployed 3 learner alpha pods.
Sometime one of learner alpha is unable to accept query requests and returns the following error: "Please retry again, server is not ready to accept requests"."
and /health?live=1 returns 200, but health returns 503.

http://localhost:8080/health, 503 Service Unavailable, "Please retry again, server is not ready to accept requests"
http://localhost:8080/health?live=1, 200

[
    {
        "instance": "alpha",
        "address": "dgraph-learner-alpha-0.dgraph-learner-alpha-headless.svc.cluster.local:7080",
        "status": "healthy",
        "group": "1",
        "version": "v25.3.1",
        "uptime": 285560,
        "lastEcho": 1777346016,
        "ee_features": [
            "backup_restore",
            "cdc"
        ],
        "max_assigned": 2206421499
    }
]

http://localhost:8080/probe/graphql , 200

{
    "status": "up",
    "schemaUpdateCounter": 0
}

alpha learner logs

E0428 03:09:49.515381       1 draft.go:1458] While retrieving snapshot, error: cannot retrieve snapshot from peer: rpc error: code = Unknown desc = operation opSnapshot is already running. Retrying...
I0428 03:09:49.515447       1 draft.go:124] Operation completed with id: opSnapshot
I0428 03:09:50.505477       1 draft.go:198] Operation started with id: opRollup
I0428 03:09:50.515906       1 draft.go:198] Operation started with id: opSnapshot
I0428 03:09:50.515973       1 draft.go:124] Operation completed with id: opRollup
E0428 03:09:50.516338       1 draft.go:1458] While retrieving snapshot, error: cannot retrieve snapshot from peer: rpc error: code = Unknown desc = operation opSnapshot is already running. Retrying...
I0428 03:09:50.516390       1 draft.go:124] Operation completed with id: opSnapshot
I0428 03:09:51.506878       1 draft.go:198] Operation started with id: opRollup
I0428 03:09:51.517209       1 draft.go:198] Operation started with id: opSnapshot
I0428 03:09:51.517313       1 draft.go:124] Operation completed with id: opRollup
E0428 03:09:51.517723       1 draft.go:1458] While retrieving snapshot, error: cannot retrieve snapshot from peer: rpc error: code = Unknown desc = operation opSnapshot is already running. Retrying...
I0428 03:09:51.517794       1 draft.go:124] Operation completed with id: opSnapshot
I0428 03:09:52.508172       1 draft.go:198] Operation started with id: opRollup
I0428 03:09:52.518652       1 draft.go:198] Operation started with id: opSnapshot
I0428 03:09:52.518756       1 draft.go:124] Operation completed with id: opRollup
E0428 03:09:52.519080       1 draft.go:1458] While retrieving snapshot, error: cannot retrieve snapshot from peer: rpc error: code = Unknown desc = operation opSnapshot is already running. Retrying...
I0428 03:09:52.519102       1 draft.go:124] Operation completed with id: opSnapshot
I0428 03:09:53.509970       1 draft.go:198] Operation started with id: opRollup
I0428 03:09:53.519332       1 draft.go:198] Operation started with id: opSnapshot
I0428 03:09:53.519421       1 draft.go:124] Operation completed with id: opRollup
E0428 03:09:53.519768       1 draft.go:1458] While retrieving snapshot, error: cannot retrieve snapshot from peer: rpc error: code = Unknown desc = operation opSnapshot is already running. Retrying...
I0428 03:09:53.519795       1 draft.go:124] Operation completed with id: opSnapshot
I0428 03:09:54.510889       1 draft.go:198] Operation started with id: opRollup

Expected behavior

alpha works normally

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

dgraph v25.3.1 with learner alpha

Additional context

Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in draft.go around the logged lines 1458, 124, and 198, then inspect how the learner handles snapshot and rollup operations. Reproduce the v25.3.1 learner-alpha condition and compare /health with /health?live=1; done means the alpha accepts requests and reports the expected health status without repeated opSnapshot errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.