ClickHouse / ClickHouse/ClickHouse
Selecting from distributed engine from inconsistent replicas
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
**Behaviour**
I have 4 servers: 2 shards 2 replicas each. clickhouse-server-22.8.4.7-1.x86_64 and build-in keeper
**Cluster config**:
Using this 1 to 1 only host names changed https://clickhouse.com/docs/en/engines/table-engines/special/distributed#distributed-clusters
Database config:
Using this 1 to 1 only db and tables changed https://github.com/go-graphite/carbon-clickhouse#clickhouse-configuration
Over tables I have distributed engine tables with cityHash64 on `Path` field. Example:
`CREATE TABLE db.data_dist on cluster '{cluster}'
(
`Path` String,
`Value` Float64,
`Time` UInt32,
`Date` Date,
`Timestamp` UInt32
)
ENGINE = Distributed('{cluster}', 'db', 'data', modulo(cityHash64(Path), toUInt32OrDefault(getMacro('number_of_shards'),toUInt32(1))));`
**number_of_shards** = 2 in clickhouse macro config
For inserting: **carbon-clickhouse** inserting to all 4 replicas of cluster
For selecting: **carbonapi** -> select using round robin from all 4 replicas of cluster using graphite-clickhouse
Graphite-clickhouse only selecting from 1 replica.
Selects and inserts are goes through distributed tables.
In this setup one metrica (with the same Path) are stored in one cluster,
Normally all working good. But if I stop 1 replica of 1 shard for 1 hour, then start it back, I have permanent lag for 1 hour between replicas in the shard and select no data for 1h data sometimes (load_balancing=random)
`SELECT * FROM system.replicas FORMAT Vertical`
for all servers have absolute_delay: 0
**Expected behavior**
Clickhouse should select data from all replicas. If there is no data, select should try other replica until failed.
- Replica with 1h delay show 1h lag
- Inconsistent replica does not participate in working until synced
**Questions**
- How I can see real lag?
- How I can disable replica from work until lag?
- How I can increase replication speed
Contributor guide
Assessment
This issue has not been assessed yet.