dgraph-io / dgraph-io/dgraph

UNAVAILABLE: Network closed for unknown reason

Open
#9,363 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Stale
Dominant language
Go
Stars
21.8k
Forks
1.6k
Avg merge
2d 5h
Merged PRs (30d)
9

Description

Describe the bug

Hi dev team, I got this bug error since update 24.1.0, when create new gcpr dgraph client and send query to Dgraph, it will the error

Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at io.grpc.Status.asRuntimeException(Status.java:532)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:814)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:494)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

there is my example query

upsert{

	query {
 
    var(func: eq(id, "buyeruid")) {

      uid_0 as uid

      is_identified_as {

        kuid_0 as uid

        ~is_identified_as {

          all_uid_0 as uid

         }

       }

     }
 
 
    var(func: uid(earliestKuid)) @filter(uid(kuid_0)) {

      isEqual_0 as uid

     }
 
 
    var(func: eq(id, "id1")) {

      uid_1 as uid

      is_identified_as {

        kuid_1 as uid

        ~is_identified_as {

          all_uid_1 as uid

         }

       }

     }
 
 
    var(func: uid(earliestKuid)) @filter(uid(kuid_1)) {

      isEqual_1 as uid

     }
 
 
    var(func: uid(kuid_0,kuid_1), orderasc: inserted_time, first: 1) {

      earliestKuid as uid

      kuid

      }


    }

  mutation @if(gt(len(uid_0), 0) AND eq(len(isEqual_0), 0)) {

		set{

			uid(all_uid_0) <is_identified_as> uid(earliestKuid) .

      uid(all_uid_0) <was_identified_as> uid(kuid_0) .

      uid(all_uid_0) <is_identified_as> uid(kuid_0) .

    }

  }

  mutation @if(gt(len(uid_0), 0) AND eq(len(isEqual_0), 0)) {

		set{

			uid(all_uid_0) <is_identified_as> uid(earliestKuid) .

      uid(all_uid_0) <was_identified_as> uid(kuid_0) .

    }

    delete{

			uid(all_uid_0) <is_identified_as> uid(kuid_0) .

    }

  }

	mutation @if(eq(len(uid_0), 0)) {

		set{

			uid(uid_0) <dgraph.type> "Identifier" .

     uid(uid_0) <id> "buyeruid" .

     uid(uid_0) <id_type> "COOKIE" .

     uid(uid_0) <id_provider> "KNOREX" .

     uid(uid_0) <data_source> "BidEvent" .

     uid(uid_0) <opted_out> "false" .

     uid(uid_0) <inserted_time> "1742900770151" .

     uid(uid_0) <expired_time> "1790334370151" .

    uid(uid_0) <is_identified_as> uid(earliestKuid) .

    }

  }


  mutation @if(gt(len(uid_1), 0) AND eq(len(isEqual_1), 0)) {

		set{

			uid(all_uid_1) <is_identified_as> uid(earliestKuid) .

			uid(all_uid_1) <was_identified_as> uid(kuid_1) .

    }

    delete{

			uid(all_uid_1) <is_identified_as> uid(kuid_1) .

    }

  }

	mutation @if(eq(len(uid_1), 0)) {

		set{

			uid(uid_1) <dgraph.type> "Identifier" .

     uid(uid_1) <id> "id1" .

     uid(uid_1) <id_type> "UID2" .

     uid(uid_1) <id_provider> "UID2" .

     uid(uid_1) <data_source> "BidEvent" .

     uid(uid_1) <opted_out> "false" .

     uid(uid_1) <inserted_time> "1742900770151" .

     uid(uid_1) <expired_time> "1790334370151" .

    uid(uid_1) <is_identified_as> uid(earliestKuid) .

    }

  }


}

this error cause both from the UI ratel and Dgraph java client. Of course we can handle it by retry, but it will slower our process.

while that alpha container will throw this error

2025-03-31 15:59:57 alpha-1          | panic: runtime error: index out of range [1] with length 0
2025-03-31 15:59:57 alpha-1          | 
2025-03-31 15:59:57 alpha-1          | goroutine 914 [running]:
2025-03-31 15:59:57 alpha-1          | github.com/hypermodeinc/dgraph/v24/edgraph.verifyUniqueWithinMutation(0x40180fa900)
2025-03-31 15:59:57 alpha-1          |  /runner/_work/dgraph/dgraph/edgraph/server.go:2161 +0x2dc
2025-03-31 15:59:57 alpha-1          | github.com/hypermodeinc/dgraph/v24/edgraph.(*Server).doMutate(0x246d000?, {0x246d000, 0x4018424690}, 0x40180fa900, 0x401843c280)
2025-03-31 15:59:57 alpha-1          |  /runner/_work/dgraph/dgraph/edgraph/server.go:584 +0xc8
2025-03-31 15:59:57 alpha-1          | github.com/hypermodeinc/dgraph/v24/edgraph.(*Server).doQuery(0x353c380, {0x246d000, 0x40187487e0}, 0x40176ed6d0)
2025-03-31 15:59:57 alpha-1          |  /runner/_work/dgraph/dgraph/edgraph/server.go:1402 +0xac4
2025-03-31 15:59:57 alpha-1          | github.com/hypermodeinc/dgraph/v24/edgraph.(*Server).QueryNoGrpc(0x353c380, {0x246d000?, 0x4018424420?}, 0x4018401100)
2025-03-31 15:59:57 alpha-1          |  /runner/_work/dgraph/dgraph/edgraph/server.go:1252 +0x250
2025-03-31 15:59:57 alpha-1          | github.com/hypermodeinc/dgraph/v24/edgraph.(*Server).Query(0x3079d90?, {0x246d000, 0x4018424420}, 0xffff71c556b8?)
2025-03-31 15:59:57 alpha-1          |  /runner/_work/dgraph/dgraph/edgraph/server.go:1218 +0x28
2025-03-31 15:59:57 alpha-1          | github.com/dgraph-io/dgo/v240/protos/api._Dgraph_Query_Handler.func1({0x246d000?, 0x4018424420?}, {0x20a7520?, 0x4018401100?})
2025-03-31 15:59:57 alpha-1          |  /home/runner/go/pkg/mod/github.com/dgraph-io/dgo/v240@v240.1.0/protos/api/api_grpc.pb.go:188 +0xd0
2025-03-31 15:59:57 alpha-1          | github.com/hypermodeinc/dgraph/v24/ee/audit.AuditRequestGRPC({0x246d000?, 0x4018424420?}, {0x20a7520?, 0x4018401100?}, 0x80?, 0x20b6b40?)
2025-03-31 15:59:57 alpha-1          |  /runner/_work/dgraph/dgraph/ee/audit/interceptor_ee.go:79 +0xd4
2025-03-31 15:59:57 alpha-1          | github.com/dgraph-io/dgo/v240/protos/api._Dgraph_Query_Handler({0x20c0bc0, 0x353c380}, {0x246d000, 0x4018424420}, 0x4018401080, 0x221cf58)
2025-03-31 15:59:57 alpha-1          |  /home/runner/go/pkg/mod/github.com/dgraph-io/dgo/v240@v240.1.0/protos/api/api_grpc.pb.go:190 +0x148
2025-03-31 15:59:57 alpha-1          | google.golang.org/grpc.(*Server).processUnaryRPC(0x40001ac000, {0x246d000, 0x40184242a0}, 0x4018233da0, 0x400b210180, 0x30a56d8, 0x0)
2025-03-31 15:59:57 alpha-1          |  /home/runner/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1400 +0xcb0
2025-03-31 15:59:57 alpha-1          | google.golang.org/grpc.(*Server).handleStream(0x40001ac000, {0x246e088, 0x4017d51040}, 0x4018233da0)
2025-03-31 15:59:57 alpha-1          |  /home/runner/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1810 +0x910
2025-03-31 15:59:57 alpha-1          | google.golang.org/grpc.(*Server).serveStreams.func2.1()
2025-03-31 15:59:57 alpha-1          |  /home/runner/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1030 +0x8c
2025-03-31 15:59:57 alpha-1          | created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 281
2025-03-31 15:59:57 alpha-1          |  /home/runner/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1041 +0x13c

To Reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

  • OS: macOS, Ubuntu
  • Language Java
  • Version 24.1.1

Additional context

Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at edgraph/server.go:2161 in verifyUniqueWithinMutation, using the supplied upsert query and the alpha panic trace as the reproduction clues. Run the issue's query against Dgraph 24.1.1 and inspect how the reported index-out-of-range failure becomes the client's gRPC UNAVAILABLE error. Done means the request no longer panics or closes the connection unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc, java
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.