store-gateway fails to start in Helm-deployed multiple microservices mode
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Describe the bug
Store-gateway pod fails to initialize with persistent Access Denied errors when attempting to synchronize Pyroscope blocks from object storage.
#### To Reproduce
Steps to reproduce the behavior:
1. Deploy Pyroscope 1.18.0 ([Helm chart pyroscope-1.3.0](https://grafana.github.io/helm-charts/)) in Kubernetes cluster
2. Configure values.yaml
```
pyroscope:
config: |
storage:
backend: s3
s3:
endpoint: "s3.example.com"
bucket_name: "example_bucket"
access_key_id: "example_ak"
secret_access_key: "example_sk"
region: "cn-shanghai"
insecure: false
extraArgs:
store-gateway.sharding-ring.replication-factor: "3"
components:
querier:
kind: Deployment
replicaCount: 3
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
cpu: 1
query-frontend:
kind: Deployment
replicaCount: 2
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
cpu: 100m
query-scheduler:
kind: Deployment
replicaCount: 2
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
cpu: 100m
distributor:
kind: Deployment
replicaCount: 2
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
cpu: 500m
ingester:
kind: StatefulSet
replicaCount: 3
terminationGracePeriodSeconds: 600
resources:
limits:
memory: 16Gi
requests:
memory: 8Gi
cpu: 1
compactor:
kind: StatefulSet
replicaCount: 3
terminationGracePeriodSeconds: 1200
persistence:
enabled: false
resources:
limits:
memory: 16Gi
requests:
memory: 8Gi
cpu: 1
store-gateway:
kind: StatefulSet
replicaCount: 3
persistence:
# The store-gateway needs not need persistent storage, but we still run it as a StatefulSet
# This is to avoid having blocks of data being
enabled: false
resources:
limits:
memory: 16Gi
requests:
memory: 8Gi
cpu: 1
readinessProbe:
# The store gateway can be configured to wait on startup for ring stability to be reached before it becomes
# ready. See the `store-gateway.sharding-ring.wait-stability-min-duration` server argument for more information.
#
# Depending on this flag and the number of tenants + blocks that need to be synced on startup, pods can take
# some time to become ready. This value can be used to ensure Kubernetes waits long enough and reduce errors.
initialDelaySeconds: 60
tenant-settings:
kind: Deployment
replicaCount: 1
resources:
limits:
memory: 4Gi
requests:
memory: 16Mi
cpu: 0.1
ad-hoc-profiles:
kind: Deployment
replicaCount: 1
resources:
limits:
memory: 4Gi
requests:
memory: 16Mi
cpu: 0.1
minio:
enabled: false
```
Restart store-gateway pod or deploy new instance
Observe logs via:
```
2026/02/04 03:35:58 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
/ _|
__ _ _ __ __ _| |_ __ _ _ __ __ _ _ __ _ _ _ __ ___ ___ ___ ___ _ __ ___
/ _' | '__/ _' | _/ _' | '_ \ / _' | | '_ \| | | | '__/ _ \/ __|/ __/ _ \| '_ \ / _ \
| (_| | | | (_| | || (_| | | | | (_| | | |_) | |_| | | | (_) \__ \ (_| (_) | |_) | __/
\__, |_| \__,_|_| \__,_|_| |_|\__,_| | .__/ \__, |_| \___/|___/\___\___/| .__/ \___|
__/ | | | __/ | | |
|___/ |_| |___/ |_|
ts=2026-02-04T03:35:58.339020073Z caller=server.go:383 level=info msg="server listening on addresses" http=[::]:4040 grpc=[::]:9095
ts=2026-02-04T03:35:58.340436869Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/admin auth=false gzip=true
ts=2026-02-04T03:35:58.342866001Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/api/swagger.json auth=false gzip=true
ts=2026-02-04T03:35:58.342908468Z caller=register_options.go:152 level=debug msg="api: registering route" methods=all prefix=/api auth=false gzip=true
ts=2026-02-04T03:35:58.34292258Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/debug/fgprof auth=false gzip=true
ts=2026-02-04T03:35:58.342933265Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET prefix=/static/ auth=false gzip=true
ts=2026-02-04T03:35:58.342957811Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET prefix=/ui auth=false gzip=true
ts=2026-02-04T03:35:58.342971221Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET prefix=/assets/ auth=false gzip=true
ts=2026-02-04T03:35:58.343193624Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/memberlist auth=false gzip=true
ts=2026-02-04T03:35:58.343658805Z caller=netutil.go:85 level=debug msg="looking for addresses" inf=[eth0] inet6enabled=false
ts=2026-02-04T03:35:58.343788859Z caller=netutil.go:102 level=debug msg="detected highest quality address" ipAddr=10.4.79.74 inf=eth0
ts=2026-02-04T03:35:58.343922173Z caller=memberlist_client.go:484 level=info msg="Using memberlist cluster label and node name" cluster_label=pyroscope-pyroscope node=pyroscope-store-gateway-2-5df8f671
ts=2026-02-04T03:35:58.3439597Z caller=log.go:245 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
ts=2026-02-04T03:35:58.344030959Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants auth=false gzip=true
ts=2026-02-04T03:35:58.344052027Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks auth=false gzip=true
ts=2026-02-04T03:35:58.344078872Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks/{block} auth=false gzip=true
ts=2026-02-04T03:35:58.34410453Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks/{block}/datasets auth=false gzip=true
ts=2026-02-04T03:35:58.34412838Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks/{block}/datasets/profiles auth=false gzip=true
ts=2026-02-04T03:35:58.344153802Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks/{block}/datasets/profiles/download auth=false gzip=true
ts=2026-02-04T03:35:58.3441897Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks/{block}/datasets/profiles/call-tree auth=false gzip=true
ts=2026-02-04T03:35:58.344220495Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks/{block}/datasets/index auth=false gzip=true
ts=2026-02-04T03:35:58.344243825Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ops/object-store/tenants/{tenant}/blocks/{block}/datasets/symbols auth=false gzip=true
ts=2026-02-04T03:35:58.344304011Z caller=tcp_transport.go:435 level=debug component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=10.4.79.74 advertisePort=7946
ts=2026-02-04T03:35:58.34436825Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/runtime_config auth=false gzip=true
ts=2026-02-04T03:35:58.344402323Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/api/v1/tenant_limits auth=true gzip=true
ts=2026-02-04T03:35:58.344567441Z caller=netutil.go:85 level=debug msg="looking for addresses" inf=[eth0] inet6enabled=false
ts=2026-02-04T03:35:58.344630335Z caller=netutil.go:102 level=debug msg="detected highest quality address" ipAddr=10.4.79.74 inf=eth0
ts=2026-02-04T03:35:58.344636988Z caller=tcp_transport.go:435 level=debug component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=10.4.79.74 advertisePort=7946
ts=2026-02-04T03:35:58.345197664Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET,POST path=/store-gateway/tenants auth=false gzip=true
ts=2026-02-04T03:35:58.345215112Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/store-gateway/tenants auth=false gzip=true
ts=2026-02-04T03:35:58.345226065Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/store-gateway/tenant/{tenant}/blocks auth=false gzip=true
ts=2026-02-04T03:35:58.345337679Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ready auth=false gzip=false
ts=2026-02-04T03:35:58.34539021Z caller=register_options.go:152 level=debug msg="api: registering route" methods=GET path=/ auth=false gzip=true
ts=2026-02-04T03:35:58.345417686Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=memberlist-kv waiting_for=server
ts=2026-02-04T03:35:58.345447926Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=runtime-config waiting_for=server
ts=2026-02-04T03:35:58.345447232Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=usage-stats waiting_for=memberlist-kv
ts=2026-02-04T03:35:58.3454467Z caller=module_service.go:82 level=info msg=starting module=server
ts=2026-02-04T03:35:58.345450222Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=store-gateway waiting_for=usage-stats
ts=2026-02-04T03:35:58.345455254Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=admin waiting_for=server
ts=2026-02-04T03:35:58.345499321Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=version waiting_for=memberlist-kv
ts=2026-02-04T03:35:58.345555667Z caller=module_service.go:82 level=info msg=starting module=admin
ts=2026-02-04T03:35:58.34555914Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2026-02-04T03:35:58.345561035Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2026-02-04T03:35:58.345622034Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=version waiting_for=server
ts=2026-02-04T03:35:58.345628293Z caller=module_service.go:82 level=info msg=starting module=version
ts=2026-02-04T03:35:58.345634474Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=usage-stats waiting_for=server
ts=2026-02-04T03:35:58.345642513Z caller=module_service.go:82 level=info msg=starting module=usage-stats
ts=2026-02-04T03:35:58.34573302Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=store-gateway waiting_for=version
ts=2026-02-04T03:35:58.345744574Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=store-gateway waiting_for=admin
ts=2026-02-04T03:35:58.345748589Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=store-gateway waiting_for=memberlist-kv
ts=2026-02-04T03:35:58.345752781Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=store-gateway waiting_for=runtime-config
ts=2026-02-04T03:35:58.345757252Z caller=module_service.go:72 level=debug msg="module waiting for initialization" module=store-gateway waiting_for=server
ts=2026-02-04T03:35:58.345762175Z caller=module_service.go:82 level=info msg=starting module=store-gateway
ts=2026-02-04T03:35:58.346398603Z caller=memberlist_client.go:628 level=info msg="memberlist fast-join starting" nodes_found=20 to_join=8
ts=2026-02-04T03:35:58.3465073Z caller=log.go:245 level=debug msg="Failed to join 10.4.79.57:7946: dial tcp 10.4.79.57:7946: connect: connection refused"
ts=2026-02-04T03:35:58.346515768Z caller=memberlist_client.go:634 level=info msg="fast-joining node failed" node=10.4.79.57:7946 err="1 error occurred:\n\t* Failed to join 10.4.79.57:7946: dial tcp 10.4.79.57:7946: connect: connection refused\n\n"
ts=2026-02-04T03:35:58.347669836Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.55.128:7946"
ts=2026-02-04T03:35:58.351811208Z caller=log.go:245 level=debug msg="Failed to join 10.4.55.192:7946: dial tcp 10.4.55.192:7946: connect: connection refused"
ts=2026-02-04T03:35:58.351827725Z caller=memberlist_client.go:634 level=info msg="fast-joining node failed" node=10.4.55.192:7946 err="1 error occurred:\n\t* Failed to join 10.4.55.192:7946: dial tcp 10.4.55.192:7946: connect: connection refused\n\n"
ts=2026-02-04T03:35:58.353058985Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.56.27:7946"
ts=2026-02-04T03:35:58.35646114Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.64.111:7946"
ts=2026-02-04T03:35:58.362057192Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.79.74:7946"
ts=2026-02-04T03:35:58.362083922Z caller=log.go:245 level=debug msg="Stream connection from=10.4.79.74:39534"
ts=2026-02-04T03:35:58.363472503Z caller=log.go:245 level=debug msg="Failed to join 10.4.79.87:7946: dial tcp 10.4.79.87:7946: connect: connection refused"
ts=2026-02-04T03:35:58.363487632Z caller=memberlist_client.go:634 level=info msg="fast-joining node failed" node=10.4.79.87:7946 err="1 error occurred:\n\t* Failed to join 10.4.79.87:7946: dial tcp 10.4.79.87:7946: connect: connection refused\n\n"
ts=2026-02-04T03:35:58.364453459Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.56.14:7946"
ts=2026-02-04T03:35:58.367270891Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.77.47:7946"
ts=2026-02-04T03:35:58.369573903Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.55.207:7946"
ts=2026-02-04T03:35:58.37306927Z caller=reporter.go:201 level=debug msg="failed to read cluster seed file" err="Forbidden path to access server."
ts=2026-02-04T03:35:58.373784459Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.55.204:7946"
ts=2026-02-04T03:35:58.377316498Z caller=memberlist_client.go:648 level=info msg="memberlist fast-join finished" joined_nodes=8 elapsed_time=32.585322ms
ts=2026-02-04T03:35:58.377509171Z caller=basic_lifecycler.go:321 level=info msg="instance not found in the ring" instance=pyroscope-store-gateway-2 ring=store-gateway
ts=2026-02-04T03:35:58.377532806Z caller=basic_lifecycler_delegates.go:63 level=info msg="not loading tokens from file, tokens file path is empty"
ts=2026-02-04T03:35:58.377759984Z caller=memberlist_client.go:666 level=info phase=startup msg="joining memberlist cluster" join_members=dns+pyroscope-memberlist.pyroscope.svc.cluster.local.:7946
ts=2026-02-04T03:35:58.378000747Z caller=gateway.go:185 level=info msg="waiting until store-gateway is JOINING in the ring"
ts=2026-02-04T03:35:58.380258959Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.56.27:7946"
ts=2026-02-04T03:35:58.384577476Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.56.25:7946"
ts=2026-02-04T03:35:58.390054295Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.64.132:7946"
ts=2026-02-04T03:35:58.392001617Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.79.74:7946"
ts=2026-02-04T03:35:58.392023638Z caller=log.go:245 level=debug msg="Stream connection from=10.4.79.74:39542"
ts=2026-02-04T03:35:58.393488637Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.64.137:7946"
ts=2026-02-04T03:35:58.394818737Z caller=log.go:245 level=debug msg="Failed to join 10.4.79.57:7946: dial tcp 10.4.79.57:7946: connect: connection refused"
ts=2026-02-04T03:35:58.395046387Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.64.111:7946"
ts=2026-02-04T03:35:58.396747844Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.78.155:7946"
ts=2026-02-04T03:35:58.398972748Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.55.207:7946"
ts=2026-02-04T03:35:58.402260746Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.78.129:7946"
ts=2026-02-04T03:35:58.404723655Z caller=log.go:245 level=debug msg="Failed to join 10.4.54.197:7946: dial tcp 10.4.54.197:7946: connect: connection refused"
ts=2026-02-04T03:35:58.405935615Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.55.204:7946"
ts=2026-02-04T03:35:58.4104667Z caller=log.go:245 level=debug msg="Failed to join 10.4.55.192:7946: dial tcp 10.4.55.192:7946: connect: connection refused"
ts=2026-02-04T03:35:58.411400909Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.56.14:7946"
ts=2026-02-04T03:35:58.415098671Z caller=log.go:245 level=debug msg="Failed to join 10.4.79.87:7946: dial tcp 10.4.79.87:7946: connect: connection refused"
ts=2026-02-04T03:35:58.416010756Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.50.77:7946"
ts=2026-02-04T03:35:58.419929309Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.77.47:7946"
ts=2026-02-04T03:35:58.421387891Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.64.139:7946"
ts=2026-02-04T03:35:58.42283043Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.64.10:7946"
ts=2026-02-04T03:35:58.425227408Z caller=log.go:245 level=debug msg="Initiating push/pull sync with: 10.4.55.128:7946"
ts=2026-02-04T03:35:58.42876711Z caller=memberlist_client.go:673 level=info phase=startup msg="joining memberlist cluster succeeded" reached_nodes=16 elapsed_time=50.981473ms
ts=2026-02-04T03:35:58.530256542Z caller=gateway.go:189 level=info msg="store-gateway is JOINING in the ring"
ts=2026-02-04T03:35:58.530341091Z caller=bucket_stores.go:171 level=info msg="synchronizing Pyroscope blocks for all users"
ts=2026-02-04T03:35:59.589618802Z caller=reporter.go:201 level=debug msg="failed to read cluster seed file" err="Forbidden path to access server."
ts=2026-02-04T03:36:02.633043786Z caller=reporter.go:201 level=debug msg="failed to read cluster seed file" err="Forbidden path to access server."
ts=2026-02-04T03:36:03.788295104Z caller=bucket_stores.go:176 level=warn msg="failed to synchronize Pyroscope blocks" err="Forbidden path to access server."
ts=2026-02-04T03:36:03.788470919Z caller=basic_lifecycler.go:262 level=info msg="ring lifecycler is shutting down" ring=store-gateway
ts=2026-02-04T03:36:03.788674572Z caller=basic_lifecycler.go:432 level=info msg="unregistering instance from ring" ring=store-gateway
ts=2026-02-04T03:36:03.788708503Z caller=basic_lifecycler.go:302 level=info msg="instance removed from the ring" ring=store-gateway
ts=2026-02-04T03:36:03.788775326Z caller=pyroscope.go:644 level=error msg="module failed" module=store-gateway error="starting module store-gateway: invalid service state: Failed, expected: Running, failure: initial blocks synchronization: Forbidden path to access server."
ts=2026-02-04T03:36:03.788796696Z caller=module_service.go:133 level=debug msg="module waiting for" module=admin waiting_for=store-gateway
ts=2026-02-04T03:36:03.788808604Z caller=module_service.go:133 level=debug msg="module waiting for" module=memberlist-kv waiting_for=store-gateway
ts=2026-02-04T03:36:03.788816085Z caller=module_service.go:110 level=debug msg=stopping module=admin
ts=2026-02-04T03:36:03.788819813Z caller=module_service.go:133 level=debug msg="module waiting for" module=memberlist-kv waiting_for=usage-stats
ts=2026-02-04T03:36:03.788832564Z caller=module_service.go:133 level=debug msg="module waiting for" module=version waiting_for=store-gateway
ts=2026-02-04T03:36:03.788850968Z caller=module_service.go:133 level=debug msg="module waiting for" module=server waiting_for=memberlist-kv
ts=2026-02-04T03:36:03.788856294Z caller=module_service.go:110 level=debug msg=stopping module=version
ts=2026-02-04T03:36:03.78885119Z caller=module_service.go:133 level=debug msg="module waiting for" module=usage-stats waiting_for=store-gateway
ts=2026-02-04T03:36:03.78886452Z caller=module_service.go:133 level=debug msg="module waiting for" module=runtime-config waiting_for=store-gateway
ts=2026-02-04T03:36:03.788880813Z caller=module_service.go:120 level=info msg="module stopped" module=admin
ts=2026-02-04T03:36:03.788883363Z caller=module_service.go:110 level=debug msg=stopping module=usage-stats
ts=2026-02-04T03:36:03.788886608Z caller=module_service.go:110 level=debug msg=stopping module=runtime-config
ts=2026-02-04T03:36:03.788895687Z caller=module_service.go:120 level=info msg="module stopped" module=version
ts=2026-02-04T03:36:03.78891522Z caller=module_service.go:120 level=info msg="module stopped" module=runtime-config
ts=2026-02-04T03:36:03.788914925Z caller=module_service.go:120 level=info msg="module stopped" module=usage-stats
ts=2026-02-04T03:36:03.788928358Z caller=module_service.go:133 level=debug msg="module waiting for" module=memberlist-kv waiting_for=version
ts=2026-02-04T03:36:03.78893407Z caller=module_service.go:110 level=debug msg=stopping module=memberlist-kv
ts=2026-02-04T03:36:03.788962176Z caller=memberlist_client.go:839 level=info msg="leaving memberlist cluster"
ts=2026-02-04T03:36:03.788970482Z caller=memberlist_client.go:850 level=info msg="waiting for locally-generated broadcast messages to be sent out" count=1 nodes=16
ts=2026-02-04T03:36:04.039859703Z caller=memberlist_client.go:850 level=info msg="waiting for locally-generated broadcast messages to be sent out" count=1 nodes=16
ts=2026-02-04T03:36:04.290550638Z caller=memberlist_client.go:850 level=info msg="waiting for locally-generated broadcast messages to be sent out" count=1 nodes=16
ts=2026-02-04T03:36:04.946056513Z caller=module_service.go:120 level=info msg="module stopped" module=memberlist-kv
ts=2026-02-04T03:36:04.946118981Z caller=module_service.go:133 level=debug msg="module waiting for" module=server waiting_for=runtime-config
ts=2026-02-04T03:36:04.946140067Z caller=module_service.go:133 level=debug msg="module waiting for" module=server waiting_for=store-gateway
ts=2026-02-04T03:36:04.946151526Z caller=module_service.go:133 level=debug msg="module waiting for" module=server waiting_for=usage-stats
ts=2026-02-04T03:36:04.946155161Z caller=module_service.go:133 level=debug msg="module waiting for" module=server waiting_for=version
ts=2026-02-04T03:36:04.946158773Z caller=module_service.go:133 level=debug msg="module waiting for" module=server waiting_for=admin
ts=2026-02-04T03:36:04.946162075Z caller=module_service.go:110 level=debug msg=stopping module=server
ts=2026-02-04T03:36:04.946328282Z caller=modules.go:769 level=info msg="server stopped"
ts=2026-02-04T03:36:04.946338132Z caller=module_service.go:120 level=info msg="module stopped" module=server
ts=2026-02-04T03:36:04.94634993Z caller=pyroscope.go:760 level=info msg="Pyroscope stopped"
failed running pyroscope: failed services
```
#### Expected behavior
✅ Transition to Running state and serve queries
#### Environment
- Infrastructure: Kubernetes v1.34.2
- Deployment tool: Helm v4.0.0
- Pyroscope version: 1.18.0 (image: grafana/pyroscope:1.18.0)
- Storage backend: TOS volcengine(S3-compatible)
- Namespace: pyroscope
#### Additional Context
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.