ClickHouse / ClickHouse/ClickHouse

Selecting from distributed engine from inconsistent replicas

Open
#51,956 0 comments 1 reaction 0 assignees View on GitHub
comp-replication external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Behaviour**
I have 4 servers: 2 shards 2 replicas each. clickhouse-server-22.8.4.7-1.x86_64 and build-in keeper
**Cluster config**:
Using this 1 to 1 only host names changed https://clickhouse.com/docs/en/engines/table-engines/special/distributed#distributed-clusters
Database config:
Using this 1 to 1 only db and tables changed https://github.com/go-graphite/carbon-clickhouse#clickhouse-configuration
Over tables I have distributed engine tables with cityHash64 on `Path` field. Example:
`CREATE TABLE db.data_dist on cluster '{cluster}'
(
`Path` String,
`Value` Float64,
`Time` UInt32,
`Date` Date,
`Timestamp` UInt32
)
ENGINE = Distributed('{cluster}', 'db', 'data', modulo(cityHash64(Path), toUInt32OrDefault(getMacro('number_of_shards'),toUInt32(1))));`
**number_of_shards** = 2 in clickhouse macro config
For inserting: **carbon-clickhouse** inserting to all 4 replicas of cluster
For selecting: **carbonapi** -> select using round robin from all 4 replicas of cluster using graphite-clickhouse
Graphite-clickhouse only selecting from 1 replica.
Selects and inserts are goes through distributed tables.
In this setup one metrica (with the same Path) are stored in one cluster,

Normally all working good. But if I stop 1 replica of 1 shard for 1 hour, then start it back, I have permanent lag for 1 hour between replicas in the shard and select no data for 1h data sometimes (load_balancing=random)

`SELECT * FROM system.replicas FORMAT Vertical`
for all servers have absolute_delay: 0

**Expected behavior**
Clickhouse should select data from all replicas. If there is no data, select should try other replica until failed.
- Replica with 1h delay show 1h lag
- Inconsistent replica does not participate in working until synced

**Questions**
- How I can see real lag?
- How I can disable replica from work until lag?
- How I can increase replication speed

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.