cockroachdb / cockroachdb/cockroach

Reconsider use of sequential keys on system tables and system-generated keys

Open
#152,012 5 comments 1 reaction 0 assignees View on GitHub
A-jobs C-enhancement O-community O-support P-3 T-jobs X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The documentation correctly discourages the use of sequential keys:

> Avoid indexing on sequential keys (e.g., [TIMESTAMP/TIMESTAMPTZ](https://www.cockroachlabs.com/docs/v25.3/timestamp) columns). Writes to indexes with sequential keys can result in [range hotspots](https://www.cockroachlabs.com/docs/v25.3/understand-hotspots#hot-range) that negatively affect performance. Instead, use [randomly generated unique IDs](https://www.cockroachlabs.com/docs/v25.3/performance-best-practices-overview#unique-id-best-practices) or [multi-column keys](https://www.cockroachlabs.com/docs/v25.3/performance-best-practices-overview#use-multi-column-primary-keys).

However, sequential keys exist in some potentially busy system tables, for example:

```
table_name | create_statement
---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
system.public.jobs | CREATE TABLE public.jobs (
| id INT8 NOT NULL DEFAULT unique_rowid(), <----------------

...
| CONSTRAINT "primary" PRIMARY KEY (id ASC),
...
| )
```

See support ticket [ZD#28257](https://cockroachdb.zendesk.com/agent/tickets/28257) for a case where this was alleged to have caused poorly distributed load due to a hot spot on a range used by the `system.jobs` table after a burst of sequential jobs IDs were created for `AUTO CREATE STATS` or `CREATE STATS` jobs.

Here is an incomplete list of other cases where the system creates sequential keys:

- Tables without a primary key declared get a PK with an ID generated by `unique_rowid()`:

```
root@localhost:26257/defaultdb> create table foo (a int);
CREATE TABLE

Time: 33ms total (execution 16ms / network 17ms)

root@localhost:26257/defaultdb> show create table foo;
table_name | create_statement
-------------+--------------------------------------------------------------
foo | CREATE TABLE public.foo (
| a INT8 NULL,
| rowid INT8 NOT VISIBLE NOT NULL DEFAULT unique_rowid(),
| CONSTRAINT foo_pkey PRIMARY KEY (rowid ASC)
| )
(1 row)
```

- Various other system tables:

```
root@localhost:26257/system> select table_catalog, table_schema, table_name, column_name from information_schema.columns where column_default = 'unique_rowid()' and table_name != 'jobs';
table_catalog | table_schema | table_name | column_name
----------------+--------------+--------------------------------+--------------
system | public | rangelog | uniqueID
system | public | web_sessions | id
system | public | table_statistics | statisticID
system | public | statement_bundle_chunks | id
system | public | statement_diagnostics_requests | id
system | public | statement_diagnostics | id
system | public | scheduled_jobs | schedule_id
(8 rows)
```

I don't have any evidence that the list above (excepting `system.jobs`) is causing a problem, but it may make sense to fix it anyway to avoid future problems at hyperscale and to avoid showing a bad example that customers might copy when creating busy user tables.

From the outside, it would seem the most straightforward fix here is to switch to `unordered_unique_rowid()`. A few years ago, we migrated a busy legacy OLTP application from `unique_rowid()` to `unordered_unique_rowid()`. Since the schema migration, we have not had any issues with hot spots caused by `unordered_unique_rowid()` and haven't perceived a need to switch to UUID keys. I would not recommend using hash-sharded indexes as a fix (though the documentation seems to prescribe this and is curiously quiet about `unordered_unique_rowid()`) due to past issues with bugs and poor performance.

Jira issue: CRDB-53591

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the system.jobs table and the other system tables listed in the issue, then inspect where unique_rowid() and unordered_unique_rowid() are used. Determine which sequential keys need reconsideration, what evidence supports a change, and how completion would be validated without introducing hotspot or compatibility regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.