scylladb / scylladb/scylla-cdc-java

Not enough replicas available for query at consistency QUORUM (5 required but only 4 alive)

Open
#61 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
32
Forks
23
Avg merge
2d 6h
Merged PRs (30d)
3

Description

My cluster is 15 nodes crossing 3 DC and five nodes for one DC,and the nodes are all UN. when i use the scylla-cdc-java tools and I get an error “Not enough replicas available for query at consistency QUORUM (5 required but only 4 alive)” ,and my keyspace RF is 3.

cqlsh> desc alternator_testcdc2;

CREATE KEYSPACE alternator_testcdc2 WITH replication = {'class': 'NetworkTopologyStrategy', 'DC01': '3', 'DC02': '3', 'DC03': '3'}  AND durable_writes = true;

CREATE TABLE alternator_testcdc2.testcdc2 (
    cdcpk text,
    cdcsk text,
    ":attrs" map<text, blob>,
    PRIMARY KEY (cdcpk, cdcsk)
) WITH CLUSTERING ORDER BY (cdcsk ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = ''
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

scylla_tags = {}

cdc = {'postimage': 'false', 'preimage': 'false', 'ttl': '86400', 'enabled': 'true', 'delta': 'full'}

CREATE TABLE alternator_testcdc2.testcdc2_scylla_cdc_log (
    "cdc$stream_id" blob,
    "cdc$time" timeuuid,
    "cdc$batch_seq_no" int,
    ":attrs" frozen<map<text, blob>>,
    "cdc$deleted_:attrs" boolean,
    "cdc$deleted_elements_:attrs" frozen<set<text>>,
    "cdc$end_of_batch" boolean,
    "cdc$operation" tinyint,
    "cdc$ttl" bigint,
    cdcpk text,
    cdcsk text,
    PRIMARY KEY ("cdc$stream_id", "cdc$time", "cdc$batch_seq_no")
) WITH CLUSTERING ORDER BY ("cdc$time" ASC, "cdc$batch_seq_no" ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'enabled': 'false', 'keys': 'NONE', 'rows_per_partition': 'NONE'}
    AND comment = 'CDC log for alternator_testcdc2.testcdc2'
    AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': '60', 'compaction_window_unit': 'MINUTES', 'expired_sstable_check_frequency_seconds': '1800'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 0
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

cqlsh>

And my cluster as followings:

[root@node-01 ~]# nodetool status
Using /etc/scylla/scylla.yaml as the config file
Datacenter: DC01
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  192.168.75.7   1.44 TB    256          ?       f1d744c8-5342-4904-98e1-71112ca776d2  DC01_Rack01
UN  192.168.75.11  1.5 TB     256          ?       ecc4ec47-f2a4-4f29-a50e-8f9f353d2a04  DC01_Rack01
UN  192.168.75.10  1.42 TB    256          ?       87e2186d-f75e-4e36-a13e-208c5719a57d  DC01_Rack01
UN  192.168.75.9   1.5 TB     256          ?       10073da2-b7d4-4362-8b55-0f319a3028fa  DC01_Rack01
UN  192.168.75.8   1.57 TB    256          ?       6c1742c8-3353-4256-9a64-0a6e126e9377  DC01_Rack01
Datacenter: DC02
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  192.168.76.7   1.44 TB    256          ?       7caa8e1d-670d-4c6e-bd65-73e9e1038cde  DC02_Rack01
UN  192.168.76.11  1.41 TB    256          ?       244b5c43-595a-4981-aaf1-ee7361f795b4  DC02_Rack01
UN  192.168.76.10  1.46 TB    256          ?       2e3eeab2-b5fa-4b55-a8b2-1cc188c9ddb1  DC02_Rack01
UN  192.168.76.9   1.53 TB    256          ?       d5c6329a-52ea-4764-b57d-235c59cff8cf  DC02_Rack01
UN  192.168.76.8   1.57 TB    256          ?       bc0879e4-df41-43b8-871a-5f9affa5cd51  DC02_Rack01
Datacenter: DC03
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  192.168.77.71  1.55 TB    256          ?       3ef50cf5-926d-4788-bf04-d7950e037514  DC03_Rack01
UN  192.168.77.70  1.57 TB    256          ?       aa9da3fc-bfb6-48c2-9e45-667177e1b61c  DC03_Rack01
UN  192.168.77.73  1.5 TB     256          ?       34f02fef-27ce-4565-bef2-0f029f29b8de  DC03_Rack01
UN  192.168.77.72  1.54 TB    256          ?       ef2786a4-0e03-439a-8383-f919bd4b9cc0  DC03_Rack01
UN  192.168.77.74  1.55 TB    256          ?       876a5b20-a061-4d24-bae0-c127fecae025  DC03_Rack01

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names the scylla-cdc-java tools and provides cqlsh schema output plus nodetool status; start by reproducing the QUORUM error with this keyspace and cluster configuration. Compare the effective replication and consistency settings, and document a confirmed cause or a clearly defined code change with a reproducible test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.