cockroachdb / cockroachdb/cockroach

changefeedccl: core changefeed fails in high data volume throughput test

Open
#134,425 2 comments 0 reactions 0 assignees View on GitHub
A-cdc branch-master C-bug T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

I observed and reproduced a core changefeed fail with a batch timestamp error while testing something unrelated.

## Reproduction:

Given `./cockroach start-single-node --insecure` running.

In 2 terminals, A and B:

A (in `./cockroach sql --insecure`):
```sql
create table stuff2 (a int primary key);
```

B:
```bash
./cockroach sql --insecure -e "EXPERIMENTAL CHANGEFEED FOR TABLE stuff2 WITH OPTIONS (updated, resolved)" | grep -v stuff2 # start a changefeed but discard the data so it doesnt spam your terminal
```

A:
```sql
insert into stuff2 (a) select i from generate_series(0, 10000) as g(i);
insert into stuff2 (a) select i from generate_series(10001, 50000) as g(i);
insert into stuff2 (a) select i from generate_series(50001, 100000) as g(i);
insert into stuff2 (a) select i from generate_series(100001, 500000) as g(i);
insert into stuff2 (a) select i from generate_series(500001, 1000000) as g(i);
insert into stuff2 (a) select i from generate_series(1000001, 5000000) as g(i);
insert into stuff2 (a) select i from generate_series(5000001, 10000000) as g(i);
```

..And then wait.. after a minute or so, the core changefeed in B will fail with:

```
(error encountered after some results were delivered)
ERROR: batch timestamp 1730910286.954309999,0 must be after replica GC threshold 1730910298.890121000,0 (r140: /Table/105/4/{3384343-5848963})
```

(table 105 here is stuff2; you can check that with `SHOW RANGES FROM DATABASE defaultdb WITH TABLES`)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.