cockroachdb / cockroachdb/cockroach

sql: import with sequence can fail due to contention

Open
#147,470 4 comments 0 reactions 0 assignees View on GitHub
A-import C-bug O-support P-3 T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We saw a case where an import into a table with a sequence column failed due to txn retry errors on the `job_info` table similar to this:
```
(XXUUU) have retried transaction: unnamed (id: efa22f8d-0890-4a6f-b15a-d64799d63deb) 2 times, most recently because of the retryable error: job 1076109732778803201: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/54/1/1076109732778803201/"legacy_progress"/1914-08-06T01:02:40.106129999Z/0 at timestamp 1748473038.880286000,2 too old; must write at or above 1748473039.289920000,2: "unnamed" meta={id=efa22f8d key=/Table/54/1/1076109732778803201/"legacy_progress" iso=Serializable pri=0.03629946 epo=1 ts=1748473039.289920000,2 min=1748473038.510288000,1 seq=0} lock=true stat=PENDING rts=1748473038.880286000,2 gul=1748473039.010288000,1 obs={n1@1748473038.510288000,1}. Terminating retry loop and returning error due to max retry limit (1). Rollback error: .
```
Here is a logictest to reproduce the failure:
```
statement ok
SET CLUSTER SETTING bulkio.import.processors_per_node = 20;

statement ok
SET CLUSTER SETTING kv.transaction.internal.max_auto_retries = 1;

statement ok
CREATE TABLE t (
id1 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id2 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id3 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id4 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id5 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id6 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id7 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id8 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id9 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
id10 INT NOT NULL GENERATED ALWAYS AS IDENTITY,
val INT
)

statement ok
INSERT INTO t (val) (SELECT i FROM generate_series(1, 10000) AS i);

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_1/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_2/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_3/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_4/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_5/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_6/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_7/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_8/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_9/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_10/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_11/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_12/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_13/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_14/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_15/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_16/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_17/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_18/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_19/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_20/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_21/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_22/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_23/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_24/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_25/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_26/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_27/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_28/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_29/' WITH DELIMITER = '|' FROM TABLE t

statement ok
EXPORT INTO CSV 'nodelocal://1/t_export_30/' WITH DELIMITER = '|' FROM TABLE t

statement ok
IMPORT INTO t CSV DATA (
'nodelocal://1/t_export_1/export*.csv',
'nodelocal://1/t_export_2/export*.csv',
'nodelocal://1/t_export_3/export*.csv',
'nodelocal://1/t_export_4/export*.csv',
'nodelocal://1/t_export_5/export*.csv',
'nodelocal://1/t_export_6/export*.csv',
'nodelocal://1/t_export_7/export*.csv',
'nodelocal://1/t_export_8/export*.csv',
'nodelocal://1/t_export_9/export*.csv',
'nodelocal://1/t_export_10/export*.csv',
'nodelocal://1/t_export_11/export*.csv',
'nodelocal://1/t_export_12/export*.csv',
'nodelocal://1/t_export_13/export*.csv',
'nodelocal://1/t_export_14/export*.csv',
'nodelocal://1/t_export_15/export*.csv',
'nodelocal://1/t_export_16/export*.csv',
'nodelocal://1/t_export_17/export*.csv',
'nodelocal://1/t_export_18/export*.csv',
'nodelocal://1/t_export_19/export*.csv',
'nodelocal://1/t_export_20/export*.csv',
'nodelocal://1/t_export_21/export*.csv',
'nodelocal://1/t_export_22/export*.csv',
'nodelocal://1/t_export_23/export*.csv',
'nodelocal://1/t_export_24/export*.csv',
'nodelocal://1/t_export_25/export*.csv',
'nodelocal://1/t_export_26/export*.csv',
'nodelocal://1/t_export_27/export*.csv',
'nodelocal://1/t_export_28/export*.csv',
'nodelocal://1/t_export_29/export*.csv',
'nodelocal://1/t_export_30/export*.csv'
) WITH DELIMITER = '|'
```
Note that the root SQL memory limit also needs to be increased to run the logic test:
```
diff --git a/pkg/sql/logictest/logic.go b/pkg/sql/logictest/logic.go
index d18d305ee7c..f53b5fabb17 100644
--- a/pkg/sql/logictest/logic.go
+++ b/pkg/sql/logictest/logic.go
@@ -1522,7 +1522,7 @@ func (t *logicTest) newCluster(

// Ensure that the server has at least 320MiB of SQL memory budget. We've
// seen cases when the default of 256MiB is insufficient.
- const maxSQLMemoryLimitLowerBound = 320 << 20 /* 320MiB */
+ const maxSQLMemoryLimitLowerBound = 320 << 23
maxSQLMemoryLimit := serverArgs.MaxSQLMemoryLimit
if maxSQLMemoryLimit <= maxSQLMemoryLimitLowerBound {
maxSQLMemoryLimit = maxSQLMemoryLimitLowerBound
```

Jira issue: CRDB-51064

Epic CRDB-62881

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.