error initialising module: store
- Dominant language
- Go
- Stars
- 28.9k
- Forks
- 4.1k
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 394
Description
**Describe the bug**
The loki service never started in SingleBinary mode.
**To Reproduce**
Steps to reproduce the behavior:
1. helm install --values single-binary-values.yaml loki grafana/loki -n logging
2. kubectl logs loki-0 --all-containers -n logging
3. kubectl logs loki-0 -c loki -n logging
4. Installation procedure based on https://sagar-srivastava.medium.com/setting-up-grafana-loki-on-kubernetes-a-simplified-guide-97fbf850ba55
**Expected behavior**
Loki starts properly on kubernetes cluster in SingleBinary mode.
**Environment:**
- Infrastructure: k3s (Client Version: v1.33.3+k3s1, Kustomize Version: v5.6.0 Server Version: v1.33.3+k3s1), xen (xcp-ng 8.3) [Kubernetes - 1 control node, 3 agent nodes]
- Deployment tool: [helm]
- The loki is connected to minio s3 storage comes with the loki installer.
**Screenshots, Promtail config, or terminal output**
Loki helm values (single-binary-values.yaml file):
```
global:
image:
registry: registry.superduper.lab
extraArgs:
- -log.level debug
- -config.expand-env=true
imagePullSecrets:
- name: registrycredentials
deploymentMode: SingleBinary
loki:
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 120
timeoutSeconds: 15
image:
registry: registry.superduper.lab
repository: registry/loki
tag: 3.4.2
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
http_server_idle_timeout: 600s
http_server_read_timeout: 600s
http_server_write_timeout: 1200s
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m
query_timeout: 300s
allow_structured_metadata: true
volume_enabled: true
commonConfig:
replication_factor: 1
instance_addr: ::0
instance_interface_names:
- enX0
- lo
ring:
kvstore:
store: inmemory
instance_addr: ::0
instance_enable_ipv6: true
instance_interface_names:
- enX0
- lo
storage:
type: s3
bucketNames:
chunks: loki-chunks
admin: loki-admin
s3:
s3forcepathstyle: true
insecure: true
schemaConfig:
configs:
- from: "2024-04-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage_config:
tsdb_shipper:
active_index_directory: /data/loki/index
cache_location: /data/loki/index_cache
cache_ttl: 24h
pattern_ingester:
enabled: true
ingester:
chunk_encoding: snappy
tracing:
enabled: true
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 1
lokiCanary:
enabled: true
image:
registry: registry.superduper.lab
repository: registry/loki-canary
tag: 3.4.2
nodeSelector:
log: loki
gateway:
image:
registry: registry.superduper.lab
repository: registry/nginx-unprivileged
tag: 1.27-alpine
pullPolicy: IfNotPresent
networkPolicy:
enabled: true
flavor: cilium
singleBinary:
replicas: 1
extraEnv:
- name: GOMEMLIMIT
value: 3750MiB
nodeSelector:
log: loki
persistence:
enabled: true
size: 4Gi
storageClass: loki-storage-class
memcached:
image:
repository: registry.superduper.lab/registry/memcached
tag: 1.6.36-alpine
memcachedExporter:
image:
repository: registry.superduper.lab/registry/memcached-exporter
tag: v0.15.0
resultsCache:
enabled: false
allocatedMemory: 256
writebackSizeLimit: 64MB
nodeSelector:
log: loki
resources:
limits:
cpu: 500m
requests:
cpu: 250m
persistence:
enabled: false
mountPath: /data/loki
chunksCache:
enabled: true
allocatedMemory: 1024
writebackSizeLimit: 10MB
nodeSelector:
log: loki
resources:
limits:
memory: 1229Mi
requests:
cpu: 500m
memory: 1229Mi
persistence:
enabled: false
mountPath: /data/loki
minio:
enabled: true
buckets:
- name: loki-chunks
policy: none
purge: false
- name: loki-ruler
policy: none
purge: false
- name: loki-admin
policy: none
purge: false
# Zero out replica counts of other deployment modes
backend:
replicas: 0
read:
replicas: 0
write:
replicas: 0
ingester:
replicas: 0
querier:
replicas: 0
queryFrontend:
replicas: 0
queryScheduler:
replicas: 0
distributor:
replicas: 0
compactor:
replicas: 0
indexGateway:
replicas: 0
bloomCompactor:
replicas: 0
bloomGateway:
replicas: 0
ruler:
enabled: false
replicas: 0
sidecar:
image:
repository: registry.superduper.lab/registry/k8s-sidecar
tag: 1.30.0
pullPolicy: IfNotPresent
rules:
logLevel: DEBUG
```
If applicable, add any output to help explain your problem.
Issue logs:
```
mkdir /data: read-only file system
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:495
main.main
/src/loki/cmd/loki/main.go:129
runtime.main
/usr/local/go/src/runtime/proc.go:272
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1700
level=info ts=2025-09-01T09:50:24.857346128Z caller=main.go:126 msg="Starting Loki" version="(version=3.4.2, branch=release-3.4.x, revision=4fa045d3)"
level=info ts=2025-09-01T09:50:24.858438197Z caller=main.go:127 msg="Loading configuration file" filename=/etc/loki/config/config.yaml
level=info ts=2025-09-01T09:50:24.866504382Z caller=server.go:351 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=error ts=2025-09-01T09:50:24.905738244Z caller=log.go:216 msg="error running loki" err="mkdir /data: read-only file system\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:495\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"
```
The local disk is available in RW mode on kubernetes node. The loki files structure exists on the disk:
ls -al /data
```
drwxrwsrwx 6 root 10001 4096 Aug 31 11:58 loki
```
ls -al /data/loki
```
drwxrwsr-x 2 10001 10001 4096 Aug 31 11:58 compactor
drwxrwsr-x 7 10001 10001 4096 Aug 31 12:49 tsdb-shipper-active
drwxrwsr-x 2 10001 10001 4096 Aug 31 11:58 tsdb-shipper-cache
drwxrws--- 2 10001 10001 4096 Aug 31 12:49 wal
```
How to set configuration '/data' path parameter in helm chart values to get rid of 'error initialising module: store' error ?
Which configuration section it concerns in helm chart values settings ?
[values.yaml](https://github.com/user-attachments/files/22076849/values.yaml)
Is it hardcoded in loki source files somewhere ?
Contributor guide
Research direction
Start with the attached values.yaml and the generated /etc/loki/config/config.yaml, then trace the store initialization reported at pkg/loki/loki.go:495. Reproduce with the shown helm install and kubectl logs commands; done means identifying the relevant /data configuration and confirming that SingleBinary Loki starts without the read-only filesystem error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100