ClickHouse / ClickHouse/graphouse

Usecase where we need to pressist data to ClickHouse the sooner the better...

Open
#73 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
261
Forks
55
PR merge metrics
No merged PRs in 30d

Description

Hi, we are a bit new to grafite... since we are using ClickHouse we also decided to use graphouse for telemetry. So we are using Grafana with ClickHouse datasource, and going directly to graphouse.data to get telemetry and what we are seeing data is actually 1 min to 5 min lag depending on volume and frequency. So I suspect is graphouse catches some data in memory using the options below (in #Mretric search and tree.. section )... if this is true what is the best setting for pushing metrics to ClickHouse no more than 30 seconds old? Is this possible? What are our options? Any advice is appreciated

We are running on 2 CPU / 8 GB RAM server

graphouse.properties
```
graphouse.allow-cold-run=false

#Clickhouse
graphouse.clickhouse.host=somehost
graphouse.clickhouse.port=8123
graphouse.clickhouse.db=graphite
graphouse.clickhouse.user=default
graphouse.clickhouse.password=ourpass
graphouse.clickhouse.compress=false

graphouse.clickhouse.data-table=data
graphouse.clickhouse.metric-tree-table=metrics

graphouse.clickhouse.socket-timeout-seconds=600
graphouse.clickhouse.query-timeout-seconds=120

graphouse.clickhouse.retention-config=

#metric server and cacher
graphouse.cacher.bind-address=0.0.0.0
graphouse.cacher.port=2003
graphouse.cacher.threads=2800
graphouse.cacher.socket-timeout-millis=42000

graphouse.cacher.queue-size=10000000
graphouse.cacher.read-batch-size=1000
graphouse.cacher.min-batch-size=10000
graphouse.cacher.max-batch-size=1000000
graphouse.cacher.min-batch-time-seconds=1
graphouse.cacher.max-batch-time-seconds=5
graphouse.cacher.max-output-threads=10000

#Http server (metric search, ping, metricData)
graphouse.http.bind-address=
graphouse.http.port=2005
graphouse.http.threads=25
graphouse.http.max-form-context-size-bytes=500000

#Mretric search and tree
graphouse.search.refresh-seconds=1
graphouse.tree.in-memory-levels=3
graphouse.tree.dir-content.cache-time-minutes=10
graphouse.tree.dir-content.cache-concurrency-level=100
graphouse.tree.dir-content.batcher.max-parallel-requests=3
graphouse.tree.dir-content.batcher.max-batch-size=2000
graphouse.tree.dir-content.batcher.aggregation-time-millis=50

#Host metrics redirect
graphouse.host-metric-redirect.enabled=false
graphouse.host-metric-redirect.dir=
graphouse.host-metric-redirect.postfixes=

#Autohide
graphouse.autohide.enabled=false
graphouse.autohide.run-delay-minutes=30
graphouse.autohide.max-values-count=200
graphouse.autohide.missing-days=7
graphouse.autohide.step=10000
graphouse.autohide.retry.count=10
graphouse.autohide.retry.wait_seconds=10
graphouse.autohide.clickhouse.query-timeout-seconds=600

#Metric validation
graphouse.metric-validation.min-length=5
graphouse.metric-validation.max-length=500
graphouse.metric-validation.min-levels=2
graphouse.metric-validation.max-levels=20
#graphouse.metric-validation.regexp=[-_0-9a-zA-Z\\.]*$
graphouse.metric-validation.regexp=.*
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.