tikv / tikv/pd

PD panic due to nil pointer dereference

Open
#3,753 7 comments 0 reactions 1 assignee Assigned to @JmPotato View on GitHub
severity/moderate status/TODO type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

```
[2021-06-08T07:36:37.659Z] [CDC:ErrGetAllStoresFailed]rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2021-06-08T07:36:37.659Z] [negroni] PANIC: runtime error: invalid memory address or nil pointer dereference
[2021-06-08T07:36:37.659Z] goroutine 42383 [running]:
[2021-06-08T07:36:37.659Z] github.com/urfave/negroni.(*Recovery).ServeHTTP.func1(0x7f5bd408ffa0, 0xc00387ddd0, 0xc000534370, 0xc00184da00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/recovery.go:159 +0xd2
[2021-06-08T07:36:37.659Z] panic(0x239e8a0, 0x3ce9210)
[2021-06-08T07:36:37.659Z] /usr/local/go/src/runtime/panic.go:965 +0x1b9
[2021-06-08T07:36:37.659Z] github.com/tikv/pd/server/cluster.(*RaftCluster).GetMetaStores(...)
[2021-06-08T07:36:37.659Z] /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/cluster/cluster.go:902
[2021-06-08T07:36:37.659Z] github.com/tikv/pd/server/api.(*storesHandler).ServeHTTP(0xc001af6e30, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc0028f8000)
[2021-06-08T07:36:37.659Z] /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/api/store.go:620 +0x50
[2021-06-08T07:36:37.659Z] github.com/tikv/pd/server/api.clusterMiddleware.Middleware.func1(0x7f5bd408ffa0, 0xc00387ddd8, 0xc0028f8000)
[2021-06-08T07:36:37.659Z] /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/api/middleware.go:43 +0x7e
[2021-06-08T07:36:37.659Z] net/http.HandlerFunc.ServeHTTP(0xc0034479b0, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc0028f8000)
[2021-06-08T07:36:37.659Z] /usr/local/go/src/net/http/server.go:2069 +0x44
[2021-06-08T07:36:37.659Z] github.com/gorilla/mux.(*Router).ServeHTTP(0xc001a943c0, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xd3
[2021-06-08T07:36:37.659Z] github.com/urfave/negroni.Wrap.func1(0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00, 0xc0040660c0)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x4d
[2021-06-08T07:36:37.659Z] github.com/urfave/negroni.HandlerFunc.ServeHTTP(0xc001bb60d8, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00, 0xc0040660c0)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x4e
[2021-06-08T07:36:37.659Z] github.com/urfave/negroni.middleware.ServeHTTP(0x2f77260, 0xc001bb60d8, 0xc001bb6120, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
[2021-06-08T07:36:37.659Z] github.com/tikv/pd/pkg/apiutil/serverapi.(*redirector).ServeHTTP(0xc000284c70, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00, 0xc0040660a0)
[2021-06-08T07:36:37.659Z] /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/pkg/apiutil/serverapi/middleware.go:98 +0x864
[2021-06-08T07:36:37.659Z] github.com/urfave/negroni.middleware.ServeHTTP(0x2f73fa0, 0xc000284c70, 0xc001bb6108, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
[2021-06-08T07:36:37.659Z] github.com/tikv/pd/pkg/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP(0xc000292b00, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00, 0xc004066080)
[2021-06-08T07:36:37.659Z] /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/pkg/apiutil/serverapi/middleware.go:54 +0xc9
[2021-06-08T07:36:37.659Z] github.com/urfave/negroni.middleware.ServeHTTP(0x2f73fc0, 0xc000292b00, 0xc001bb60f0, 0x7f5bd408ffa0, 0xc00387ddd8, 0xc00184de00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
[2021-06-08T07:36:37.659Z] github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc001ba6db0, 0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184de00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0xf1
[2021-06-08T07:36:37.659Z] github.com/gorilla/mux.(*Router).ServeHTTP(0xc001a94300, 0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184dc00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xd3
[2021-06-08T07:36:37.659Z] github.com/gorilla/mux.(*Router).ServeHTTP(0xc001a94240, 0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184da00)
[2021-06-08T07:36:37.659Z] /nfs/cache/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xd3
[2021-06-08T07:36:37.660Z] github.com/urfave/negroni.Wrap.func1(0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184da00, 0xc004066060)
[2021-06-08T07:36:37.660Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x4d
[2021-06-08T07:36:37.660Z] github.com/urfave/negroni.HandlerFunc.ServeHTTP(0xc001bb6618, 0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184da00, 0xc004066060)
[2021-06-08T07:36:37.660Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x4e
[2021-06-08T07:36:37.660Z] github.com/urfave/negroni.middleware.ServeHTTP(0x2f77260, 0xc001bb6618, 0xc001bb6648, 0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184da00)
[2021-06-08T07:36:37.660Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
[2021-06-08T07:36:37.660Z] github.com/urfave/negroni.(*Recovery).ServeHTTP(0xc000534370, 0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184da00, 0xc004066040)
[2021-06-08T07:36:37.660Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/recovery.go:193 +0x8c
[2021-06-08T07:36:37.660Z] github.com/urfave/negroni.middleware.ServeHTTP(0x2f74260, 0xc000534370, 0xc001bb6630, 0x7f5bd408ffa0, 0xc00387ddd0, 0xc00184da00)
[2021-06-08T07:36:37.660Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x9c
[2021-06-08T07:36:37.660Z] github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc00017d7d0, 0x2f9f060, 0xc00216aee0, 0xc00184da00)
[2021-06-08T07:36:37.660Z] /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0xf1
[2021-06-08T07:36:37.660Z] net/http.(*ServeMux).ServeHTTP(0xc0022b7600, 0x2f9f060, 0xc00216aee0, 0xc00184da00)
[2021-06-08T07:36:37.660Z] /usr/local/go/src/net/http/server.go:2448 +0x1ad
[2021-06-08T07:36:37.660Z] go.etcd.io/etcd/embed.(*accessController).ServeHTTP(0xc000640d80, 0x2f9f060, 0xc00216aee0, 0xc00184da00)
[2021-06-08T07:36:37.660Z] /nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/embed/serve.go:359 +0x1b9
[2021-06-08T07:36:37.660Z] net/http.serverHandler.ServeHTTP(0xc001e10380, 0x2f9f060, 0xc00216aee0, 0xc00184da00)
[2021-06-08T07:36:37.660Z] /usr/local/go/src/net/http/server.go:2887 +0xa3
[2021-06-08T07:36:37.660Z] net/http.(*conn).serve(0xc000166a00, 0x2fa70d8, 0xc0034f9600)
[2021-06-08T07:36:37.660Z] /usr/local/go/src/net/http/server.go:1952 +0x8cd
[2021-06-08T07:36:37.660Z] created by net/http.(*Server).Serve
[2021-06-08T07:36:37.660Z] /usr/local/go/src/net/http/server.go:3013 +0x39b
[2021-06-08T07:36:37.918Z] [2021/06/08 15:36:37.651 +08:00] [WARN] [retry_interceptor.go:61] ["retrying of unary invoker failed"] [target=endpoint://client-f93c591b-045a-4879-872d-c7883e5a35d0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"]
```
See https://ci.pingcap.net/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/4494/pipeline/#step-301-log-213

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.