grafana / grafana/loki

Rate limit misconf loki ingester

Open
#17,536 1 comment 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Go
Stars
28.9k
Forks
4.1k
Avg merge
1d 16h
Merged PRs (30d)
395

Description

**Describe the bug**
Hi there,
I'm seeing a storage behavior, I'm getting 429 even after increasing my rate limit:

`
level=error ts=2025-05-01T06:05:56.600068163Z caller=manager.go:50 component=distributor path=write msg="write operation failed" details="Ingestion rate limit exceeded for user application (limit: 2396745 bytes/sec) while attempting to ingest '7694' lines totaling '819178' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased" org_id=application
`

The limit is bigger than the current ingested value

My Loki Config:

```
helmCharts:
- name: loki
includeCRDs: true
version: 6.28.0
repo: https://grafana.github.io/helm-charts
valuesFile: values.yml
releaseName: xx
namespace: loki

```

```
monitoring:
dashboards:
enabled: true
serviceMonitor:
enabled: true
gateway:
autoscaling:
enabled: true
minReplicas: 5
maxReplicas: 10
service:
type: LoadBalancer
annotations:
networking.gke.io/load-balancer-type: "Internal"
basicAuth:
enabled: false
username: asgard
password: P@s5w0rD

test:
enabled: false
lokiCanary:
enabled: false

loki:
querier:
multi_tenant_queries_enabled: true
limits_config:
per_stream_rate_limit: 20MB
ingestion_burst_size_mb: 100
per_stream_rate_limit_burst: 20MB
ingestion_rate_mb: 16
allow_structured_metadata: false
max_global_streams_per_user: 100000
max_line_size: 5012KB
max_label_names_per_series: 30
pattern_ingester:
enabled: true
auth_enabled: true
minio:
enabled: false
storage:
type: gcs
bucketNames:
chunks: "prod-loki-store"
ruler: "prod-loki-store"
admin: "prod-loki-store"
object_store:
gcs:
bucket_name: "prod-loki-store"
service_account: lokisa
useTestSchema: false
schemaConfig:
configs:
- from: 2024-04-01
object_store: gcs
store: boltdb-shipper
schema: v11
index:
prefix: index_
period: 24h

deploymentMode: "Distributed"

minio:
enabled: false

querier:
replicas: 6
maxUnavailable: 2
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
queryFrontend:
replicas: 6
maxUnavailable: 1
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
queryScheduler:
replicas: 6
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
distributor:
replicas: 7
maxUnavailable: 2
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
compactor:
replicas: 3
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
indexGateway:
replicas: 4
maxUnavailable: 1
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
chunksCache:
enabled: false
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"

ingester:
replicas: 7
zoneAwareReplication:
enabled: false
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"

resultsCache:
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"

bloomPlanner:
replicas: 0
bloomBuilder:
replicas: 0
bloomGateway:
replicas: 0

read:
replicas: 0
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
backend:
replicas: 0
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"
write:
replicas: 0
nodeSelector:
type: monitoring
tolerations:
- key: "monitoring-node-pool"
operator: "Exists"

singleBinary:
replicas: 0

serviceAccount:
create: true
name: lokisa
annotations:
iam.gke.io/gcp-service-account: lokisa@mysa

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.