Rate limit misconf loki ingester
- Dominant language
- Go
- Stars
- 28.9k
- Forks
- 4.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 395
Description
**Describe the bug**
Hi there,
I'm seeing a storage behavior, I'm getting 429 even after increasing my rate limit:
`
level=error ts=2025-05-01T06:05:56.600068163Z caller=manager.go:50 component=distributor path=write msg="write operation failed" details="Ingestion rate limit exceeded for user application (limit: 2396745 bytes/sec) while attempting to ingest '7694' lines totaling '819178' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased" org_id=application
`
The limit is bigger than the current ingested value
My Loki Config:
```
helmCharts:
- name: loki
includeCRDs: true
version: 6.28.0
repo: https://grafana.github.io/helm-charts
valuesFile: values.yml
releaseName: xx
namespace: loki
```
```
monitoring:
dashboards:
enabled: true
serviceMonitor:
enabled: true
gateway:
autoscaling:
enabled: true
minReplicas: 5
maxReplicas: 10
service:
type: LoadBalancer
annotations:
networking.gke.io/load-balancer-type: "Internal"
basicAuth:
enabled: false
username: asgard
password: P@s5w0rD
test:
enabled: false
lokiCanary:
enabled: false
loki:
querier:
multi_tenant_queries_enabled: true
limits_config:
per_stream_rate_limit: 20MB
ingestion_burst_size_mb: 100
per_stream_rate_limit_burst: 20MB
ingestion_rate_mb: 16
allow_structured_metadata: false
max_global_streams_per_user: 100000
max_line_size: 5012KB
max_label_names_per_series: 30
pattern_ingester:
enabled: true
auth_enabled: true
minio:
enabled: false
storage:
type: gcs
bucketNames:
chunks: "prod-loki-store"
ruler: "prod-loki-store"
admin: "prod-loki-store"
object_store:
gcs:
bucket_name: "prod-loki-store"
service_account: lokisa
useTestSchema: false
schemaConfig:
configs:
- from: 2024-04-01
object_store: gcs
store: boltdb-shipper
schema: v11
index:
prefix: index_
period: 24h
deploymentMode: "Distributed"
minio:
enabled: false
querier:
replicas: 6
maxUnavailable: 2
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
queryFrontend:
replicas: 6
maxUnavailable: 1
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
queryScheduler:
replicas: 6
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
distributor:
replicas: 7
maxUnavailable: 2
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
compactor:
replicas: 3
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
indexGateway:
replicas: 4
maxUnavailable: 1
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
chunksCache:
enabled: false
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
ingester:
replicas: 7
zoneAwareReplication:
enabled: false
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
resultsCache:
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
bloomPlanner:
replicas: 0
bloomBuilder:
replicas: 0
bloomGateway:
replicas: 0
read:
replicas: 0
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
backend:
replicas: 0
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
write:
replicas: 0
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
singleBinary:
replicas: 0
serviceAccount:
create: true
name: lokisa
annotations:
iam.gke.io/gcp-service-account: lokisa@mysa
```
Contributor guide
Assessment
This issue has not been assessed yet.