cockroachdb / cockroachdb/cockroach
changefeedccl: core changefeed fails in high data volume throughput test
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
I observed and reproduced a core changefeed fail with a batch timestamp error while testing something unrelated.
## Reproduction:
Given `./cockroach start-single-node --insecure` running.
In 2 terminals, A and B:
A (in `./cockroach sql --insecure`):
```sql
create table stuff2 (a int primary key);
```
B:
```bash
./cockroach sql --insecure -e "EXPERIMENTAL CHANGEFEED FOR TABLE stuff2 WITH OPTIONS (updated, resolved)" | grep -v stuff2 # start a changefeed but discard the data so it doesnt spam your terminal
```
A:
```sql
insert into stuff2 (a) select i from generate_series(0, 10000) as g(i);
insert into stuff2 (a) select i from generate_series(10001, 50000) as g(i);
insert into stuff2 (a) select i from generate_series(50001, 100000) as g(i);
insert into stuff2 (a) select i from generate_series(100001, 500000) as g(i);
insert into stuff2 (a) select i from generate_series(500001, 1000000) as g(i);
insert into stuff2 (a) select i from generate_series(1000001, 5000000) as g(i);
insert into stuff2 (a) select i from generate_series(5000001, 10000000) as g(i);
```
..And then wait.. after a minute or so, the core changefeed in B will fail with:
```
(error encountered after some results were delivered)
ERROR: batch timestamp 1730910286.954309999,0 must be after replica GC threshold 1730910298.890121000,0 (r140: /Table/105/4/{3384343-5848963})
```
(table 105 here is stuff2; you can check that with `SHOW RANGES FROM DATABASE defaultdb WITH TABLES`)
Contributor guide
Assessment
This issue has not been assessed yet.