cockroachdb / cockroachdb/cockroach
Reconsider use of sequential keys on system tables and system-generated keys
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The documentation correctly discourages the use of sequential keys:
> Avoid indexing on sequential keys (e.g., [TIMESTAMP/TIMESTAMPTZ](https://www.cockroachlabs.com/docs/v25.3/timestamp) columns). Writes to indexes with sequential keys can result in [range hotspots](https://www.cockroachlabs.com/docs/v25.3/understand-hotspots#hot-range) that negatively affect performance. Instead, use [randomly generated unique IDs](https://www.cockroachlabs.com/docs/v25.3/performance-best-practices-overview#unique-id-best-practices) or [multi-column keys](https://www.cockroachlabs.com/docs/v25.3/performance-best-practices-overview#use-multi-column-primary-keys).
However, sequential keys exist in some potentially busy system tables, for example:
```
table_name | create_statement
---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
system.public.jobs | CREATE TABLE public.jobs (
| id INT8 NOT NULL DEFAULT unique_rowid(), <----------------
...
| CONSTRAINT "primary" PRIMARY KEY (id ASC),
...
| )
```
See support ticket [ZD#28257](https://cockroachdb.zendesk.com/agent/tickets/28257) for a case where this was alleged to have caused poorly distributed load due to a hot spot on a range used by the `system.jobs` table after a burst of sequential jobs IDs were created for `AUTO CREATE STATS` or `CREATE STATS` jobs.
Here is an incomplete list of other cases where the system creates sequential keys:
- Tables without a primary key declared get a PK with an ID generated by `unique_rowid()`:
```
root@localhost:26257/defaultdb> create table foo (a int);
CREATE TABLE
Time: 33ms total (execution 16ms / network 17ms)
root@localhost:26257/defaultdb> show create table foo;
table_name | create_statement
-------------+--------------------------------------------------------------
foo | CREATE TABLE public.foo (
| a INT8 NULL,
| rowid INT8 NOT VISIBLE NOT NULL DEFAULT unique_rowid(),
| CONSTRAINT foo_pkey PRIMARY KEY (rowid ASC)
| )
(1 row)
```
- Various other system tables:
```
root@localhost:26257/system> select table_catalog, table_schema, table_name, column_name from information_schema.columns where column_default = 'unique_rowid()' and table_name != 'jobs';
table_catalog | table_schema | table_name | column_name
----------------+--------------+--------------------------------+--------------
system | public | rangelog | uniqueID
system | public | web_sessions | id
system | public | table_statistics | statisticID
system | public | statement_bundle_chunks | id
system | public | statement_diagnostics_requests | id
system | public | statement_diagnostics | id
system | public | scheduled_jobs | schedule_id
(8 rows)
```
I don't have any evidence that the list above (excepting `system.jobs`) is causing a problem, but it may make sense to fix it anyway to avoid future problems at hyperscale and to avoid showing a bad example that customers might copy when creating busy user tables.
From the outside, it would seem the most straightforward fix here is to switch to `unordered_unique_rowid()`. A few years ago, we migrated a busy legacy OLTP application from `unique_rowid()` to `unordered_unique_rowid()`. Since the schema migration, we have not had any issues with hot spots caused by `unordered_unique_rowid()` and haven't perceived a need to switch to UUID keys. I would not recommend using hash-sharded indexes as a fix (though the documentation seems to prescribe this and is curiously quiet about `unordered_unique_rowid()`) due to past issues with bugs and poor performance.
Jira issue: CRDB-53591
Contributor guide
Research direction
Start by reviewing the system.jobs table and the other system tables listed in the issue, then inspect where unique_rowid() and unordered_unique_rowid() are used. Determine which sequential keys need reconsideration, what evidence supports a change, and how completion would be validated without introducing hotspot or compatibility regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100