memberlist error on EKS
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Describe the bug
We installed pyroscope on EKS 1.28 but pyroscope-0 pod got CrashLoopBackOff
Logs from the pod:
failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
#### To Reproduce
Steps to reproduce the behavior:
Install pyroscope on EKS cluster:
helm upgrade --install pyroscope grafana/pyroscope
#### Expected behavior
The chart is installed successfully
#### Environment
EKS 1.28
pyroscope 1.2.0
Chart version: latest
#### Additional Context
Full logs:
ts=2023-11-24T09:26:00.950349984Z caller=server.go:338 level=info msg="server listening on addresses" http=[::]:4040 grpc=[::]:9095
ts=2023-11-24T09:26:00.951878623Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/admin auth=false gzip=true
ts=2023-11-24T09:26:00.953432563Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/api/swagger.json auth=false gzip=true
ts=2023-11-24T09:26:00.953550529Z caller=api.go:96 level=debug msg="api: registering route" methods=GET,POST,PUT,DELETE,HEAD,OPTIONS prefix=/api auth=false gzip=true
ts=2023-11-24T09:26:00.953634534Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/debug/fgprof auth=false gzip=true
ts=2023-11-24T09:26:00.953681355Z caller=api.go:96 level=debug msg="api: registering route" methods=GET prefix=/static/ auth=false gzip=true
ts=2023-11-24T09:26:00.953747424Z caller=api.go:96 level=debug msg="api: registering route" methods=GET prefix=/ui auth=false gzip=true
ts=2023-11-24T09:26:00.953778149Z caller=api.go:96 level=debug msg="api: registering route" methods=GET prefix=/assets/ auth=false gzip=true
ts=2023-11-24T09:26:00.954277414Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/memberlist auth=false gzip=true
ts=2023-11-24T09:26:00.955375063Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/runtime_config auth=false gzip=true
ts=2023-11-24T09:26:00.955440262Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/api/v1/tenant_limits auth=true gzip=true
ts=2023-11-24T09:26:00.956918612Z caller=api.go:91 level=debug msg="api: registering route" methods=GET,POST path=/ring auth=false gzip=true
ts=2023-11-24T09:26:00.957603841Z caller=netutil.go:85 level=debug component=overrides-exporter msg="looking for addresses" inf="[eth0 en0]" inet6enabled=true
ts=2023-11-24T09:26:00.957851814Z caller=netutil.go:102 level=debug component=overrides-exporter msg="detected highest quality address" ipAddr=192.20.8.104 inf=eth0
ts=2023-11-24T09:26:00.957965038Z caller=netutil.go:90 level=warn component=overrides-exporter msg="error getting addresses for interface" inf=en0 err="route ip+net: no such network interface"
ts=2023-11-24T09:26:00.957990697Z caller=netutil.go:118 level=debug component=overrides-exporter msg="detected IP address after looking up all configured interface names" ipAddr=192.20.8.104
ts=2023-11-24T09:26:00.958795287Z caller=api.go:91 level=debug msg="api: registering route" methods=GET,POST path=/overrides-exporter/ring auth=false gzip=true
ts=2023-11-24T09:26:00.959572013Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/pyroscope/render auth=true gzip=true
ts=2023-11-24T09:26:00.959621881Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/pyroscope/render-diff auth=true gzip=true
ts=2023-11-24T09:26:00.959656113Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/pyroscope/label-values auth=true gzip=true
ts=2023-11-24T09:26:00.959938911Z caller=memberlist_client.go:434 level=info msg="Using memberlist cluster label and node name" cluster_label=pyroscope-pyroscope node=pyroscope-0-3c39d07a
ts=2023-11-24T09:26:00.960274148Z caller=log.go:245 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
ts=2023-11-24T09:26:00.961130333Z caller=netutil.go:85 level=debug component=distributor msg="looking for addresses" inf="[eth0 en0]" inet6enabled=false
ts=2023-11-24T09:26:00.961299582Z caller=netutil.go:102 level=debug component=distributor msg="detected highest quality address" ipAddr=192.20.8.104 inf=eth0
ts=2023-11-24T09:26:00.961735344Z caller=api.go:91 level=debug msg="api: registering route" methods=POST path=/ingest auth=true gzip=true
ts=2023-11-24T09:26:00.961877238Z caller=api.go:91 level=debug msg="api: registering route" methods=POST path=/pyroscope/ingest auth=true gzip=true
ts=2023-11-24T09:26:00.962079928Z caller=api.go:91 level=debug msg="api: registering route" methods=GET,POST path=/distributor/ring auth=false gzip=true
ts=2023-11-24T09:26:00.962552243Z caller=worker.go:113 level=info component=querier-worker msg="Starting querier worker connected to query-scheduler" scheduler=
ts=2023-11-24T09:26:00.963127953Z caller=netutil.go:85 level=debug msg="looking for addresses" inf="[eth0 en0]" inet6enabled=false
ts=2023-11-24T09:26:00.963319902Z caller=netutil.go:102 level=debug msg="detected highest quality address" ipAddr=192.20.8.104 inf=eth0
ts=2023-11-24T09:26:00.964416091Z caller=netutil.go:85 level=debug component=scheduler msg="looking for addresses" inf="[eth0 en0]" inet6enabled=false
ts=2023-11-24T09:26:00.964681918Z caller=netutil.go:102 level=debug component=scheduler msg="detected highest quality address" ipAddr=192.20.8.104 inf=eth0
ts=2023-11-24T09:26:00.965184888Z caller=api.go:91 level=debug msg="api: registering route" methods=GET path=/ready auth=false gzip=false
ts=2023-11-24T09:26:00.965396484Z caller=api.go:96 level=debug msg="api: registering route" methods=GET prefix=/ auth=false gzip=true
ts=2023-11-24T09:26:00.965470428Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=querier waiting_for=memberlist-kv
ts=2023-11-24T09:26:00.965732855Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester waiting_for=server
ts=2023-11-24T09:26:00.96573339Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=overrides-exporter waiting_for=server
ts=2023-11-24T09:26:00.965772069Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=query-frontend waiting_for=memberlist-kv
ts=2023-11-24T09:26:00.965864073Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=distributor waiting_for=memberlist-kv
ts=2023-11-24T09:26:00.965909913Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=query-scheduler waiting_for=usage-stats
ts=2023-11-24T09:26:00.966266893Z caller=module_service.go:82 level=info msg=initialising module=server
ts=2023-11-24T09:26:00.966443568Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=memberlist-kv waiting_for=server
ts=2023-11-24T09:26:00.966474354Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=usage-stats waiting_for=memberlist-kv
ts=2023-11-24T09:26:00.966604989Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=runtime-config waiting_for=server
ts=2023-11-24T09:26:00.966693457Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
ts=2023-11-24T09:26:00.966857Z caller=module_service.go:82 level=info msg=initialising module=runtime-config
ts=2023-11-24T09:26:00.967152324Z caller=module_service.go:82 level=info msg=initialising module=memberlist-kv
ts=2023-11-24T09:26:00.967285809Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=overrides-exporter waiting_for=memberlist-kv
ts=2023-11-24T09:26:00.967378467Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=ingester waiting_for=usage-stats
ts=2023-11-24T09:26:00.967514781Z caller=phlare.go:413 level=error msg="module failed" module=memberlist-kv error="invalid service state: Failed, expected: Running, failure: service &{0xc000941c20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 pyroscope-pyroscope false [dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false false { false }} [{ringDesc 0xeb88e0} {}]} 0xc0005e1000 0x3d27bc0 0xc00075b2d0 {[] {[] 0}} {0 0} map[] map[ringDesc:{ringDesc 0xeb88e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000102360 0xc000947500 0xc000947620 0xc000947740 0xc000947860 0xc000947bc0 0xc000947980 0xc000947ce0 0xc000947aa0 0xc00094d780 0xc00094d7c0 0xc000947f80 0xc0009680c0 0xc000968300 0xc0009681e0 0xc0000757c8 0xc0009683c0 0xc0000757b8 0xc0000757c0 0xc00094d900 0xc00094d940 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
ts=2023-11-24T09:26:00.967645114Z caller=module_service.go:127 level=debug msg="module waiting for" module=runtime-config waiting_for=ingester
ts=2023-11-24T09:26:00.96774781Z caller=phlare.go:413 level=error msg="module failed" module=overrides-exporter error="failed to start overrides-exporter, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000941c20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 pyroscope-pyroscope false [dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false false { false }} [{ringDesc 0xeb88e0} {}]} 0xc0005e1000 0x3d27bc0 0xc00075b2d0 {[] {[] 0}} {0 0} map[] map[ringDesc:{ringDesc 0xeb88e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000102360 0xc000947500 0xc000947620 0xc000947740 0xc000947860 0xc000947bc0 0xc000947980 0xc000947ce0 0xc000947aa0 0xc00094d780 0xc00094d7c0 0xc000947f80 0xc0009680c0 0xc000968300 0xc0009681e0 0xc0000757c8 0xc0009683c0 0xc0000757b8 0xc0000757c0 0xc00094d900 0xc00094d940 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
ts=2023-11-24T09:26:00.967804952Z caller=phlare.go:413 level=error msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000941c20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 pyroscope-pyroscope false [dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false false { false }} [{ringDesc 0xeb88e0} {}]} 0xc0005e1000 0x3d27bc0 0xc00075b2d0 {[] {[] 0}} {0 0} map[] map[ringDesc:{ringDesc 0xeb88e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000102360 0xc000947500 0xc000947620 0xc000947740 0xc000947860 0xc000947bc0 0xc000947980 0xc000947ce0 0xc000947aa0 0xc00094d780 0xc00094d7c0 0xc000947f80 0xc0009680c0 0xc000968300 0xc0009681e0 0xc0000757c8 0xc0009683c0 0xc0000757b8 0xc0000757c0 0xc00094d900 0xc00094d940 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
ts=2023-11-24T09:26:00.967968055Z caller=phlare.go:413 level=error msg="module failed" module=usage-stats error="failed to start usage-stats, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000941c20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 pyroscope-pyroscope false [dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false false { false }} [{ringDesc 0xeb88e0} {}]} 0xc0005e1000 0x3d27bc0 0xc00075b2d0 {[] {[] 0}} {0 0} map[] map[ringDesc:{ringDesc 0xeb88e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000102360 0xc000947500 0xc000947620 0xc000947740 0xc000947860 0xc000947bc0 0xc000947980 0xc000947ce0 0xc000947aa0 0xc00094d780 0xc00094d7c0 0xc000947f80 0xc0009680c0 0xc000968300 0xc0009681e0 0xc0000757c8 0xc0009683c0 0xc0000757b8 0xc0000757c0 0xc00094d900 0xc00094d940 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
ts=2023-11-24T09:26:00.968067126Z caller=phlare.go:413 level=error msg="module failed" module=distributor error="failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000941c20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 pyroscope-pyroscope false [dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false false {
false }} [{ringDesc 0xeb88e0} {}]} 0xc0005e1000 0x3d27bc0 0xc00075b2d0 {[] {[] 0}} {0 0} map[] map[ringDesc:{ringDesc 0xeb88e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000102360 0xc000947500 0xc000947620 0xc000947740 0xc000947860 0xc000947bc0 0xc000947980 0xc000947ce0 0xc000947aa0 0xc00094d780 0xc00094d7c0 0xc000947f80 0xc0009680c0 0xc000968300 0xc0009681e0 0xc0000757c8 0xc0009683c0 0xc0000757b8 0xc0000757c0 0xc00094d900 0xc00094d940 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
ts=2023-11-24T09:26:00.968183358Z caller=phlare.go:413 level=error msg="module failed" module=query-frontend error="failed to start query-frontend, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000941c20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 pyroscope-pyroscope false [dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false false { false }} [{ringDesc 0xeb88e0} {}]} 0xc0005e1000 0x3d27bc0 0xc00075b2d0 {[] {[] 0}} {0 0} map[] map[ringDesc:{ringDesc 0xeb88e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000102360 0xc000947500 0xc000947620 0xc000947740 0xc000947860 0xc000947bc0 0xc000947980 0xc000947ce0 0xc000947aa0 0xc00094d780 0xc00094d7c0 0xc000947f80 0xc0009680c0 0xc000968300 0xc0009681e0 0xc0000757c8 0xc0009683c0 0xc0000757b8 0xc0000757c0 0xc00094d900 0xc00094d940 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
ts=2023-11-24T09:26:00.968284761Z caller=phlare.go:413 level=error msg="module failed" module=querier error="failed to start querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000941c20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 pyroscope-pyroscope false [dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946] 1000000000 60000000000 10 false 0 300000000000 20000000000 0 {[] 7946 2000000000 5000000000 false false { false }} [{ringDesc 0xeb88e0} {}]} 0xc0005e1000 0x3d27bc0 0xc00075b2d0 {[] {[] 0}} {0 0} map[] map[ringDesc:{ringDesc 0xeb88e0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc000102360 0xc000947500 0xc000947620 0xc000947740 0xc000947860 0xc000947bc0 0xc000947980 0xc000947ce0 0xc000947aa0 0xc00094d780 0xc00094d7c0 0xc000947f80 0xc0009680c0 0xc000968300 0xc0009681e0 0xc0000757c8 0xc0009683c0 0xc0000757b8 0xc0000757c0 0xc00094d900 0xc00094d940 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
ts=2023-11-24T09:26:00.968414939Z caller=phlare.go:413 level=error msg="module failed" module=query-scheduler error="failed to start query-scheduler, because it depends on module usage-stats, which has failed: context canceled"
ts=2023-11-24T09:26:00.968539633Z caller=phlare.go:413 level=error msg="module failed" module=ingester error="failed to start ingester, because it depends on module usage-stats, which has failed: context canceled"
ts=2023-11-24T09:26:00.968649474Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=usage-stats
ts=2023-11-24T09:26:00.96867243Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=memberlist-kv
ts=2023-11-24T09:26:00.96868464Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=overrides-exporter
ts=2023-11-24T09:26:00.968802152Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=querier
ts=2023-11-24T09:26:00.968879949Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=query-scheduler
ts=2023-11-24T09:26:00.968967779Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=ring
ts=2023-11-24T09:26:00.969044003Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=runtime-config
ts=2023-11-24T09:26:00.969145962Z caller=module_service.go:127 level=debug msg="module waiting for" module=runtime-config waiting_for=overrides-exporter
ts=2023-11-24T09:26:00.969224335Z caller=module_service.go:127 level=debug msg="module waiting for" module=runtime-config waiting_for=querier
ts=2023-11-24T09:26:00.969309822Z caller=module_service.go:127 level=debug msg="module waiting for" module=runtime-config waiting_for=query-frontend
ts=2023-11-24T09:26:00.969387931Z caller=module_service.go:127 level=debug msg="module waiting for" module=runtime-config waiting_for=query-scheduler
ts=2023-11-24T09:26:00.969457181Z caller=module_service.go:127 level=debug msg="module waiting for" module=runtime-config waiting_for=distributor
ts=2023-11-24T09:26:00.96947529Z caller=module_service.go:104 level=debug msg=stopping module=runtime-config
ts=2023-11-24T09:26:00.969642294Z caller=module_service.go:114 level=info msg="module stopped" module=runtime-config
ts=2023-11-24T09:26:00.969757876Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=distributor
ts=2023-11-24T09:26:00.969780335Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=ingester
ts=2023-11-24T09:26:00.969882011Z caller=module_service.go:127 level=debug msg="module waiting for" module=server waiting_for=query-frontend
ts=2023-11-24T09:26:00.969969594Z caller=module_service.go:104 level=debug msg=stopping module=server
ts=2023-11-24T09:26:00.970314905Z caller=modules.go:642 level=info msg="server stopped"
ts=2023-11-24T09:26:00.970337802Z caller=module_service.go:114 level=info msg="module stopped" module=server
ts=2023-11-24T09:26:00.970466352Z caller=phlare.go:487 level=info msg="Pyroscope stopped"
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported `helm upgrade --install pyroscope grafana/pyroscope` deployment on EKS 1.28 and inspect the `pyroscope-0` logs around `memberlist-kv` startup. Trace the memberlist configuration and chart values involved in selecting the advertise address; done means the chart installs successfully and the pod no longer enters CrashLoopBackOff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, helm, kubernetes
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100