cockroachdb / cockroachdb/cockroach

pg_catalog introspection queries are ~100-1000x slower than PostgreSQL during execution of Django test suite (when hundreds of tables are present)

Open
#173,076 3 comments 0 reactions 1 assignee Claimed by @rafiss View on GitHub
A-many-descriptors C-bug O-community T-sql-foundations X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Previous optimization work was done in https://github.com/cockroachdb/cockroach/issues/95068. This follow up was created with the help of Claude Code. My expertise is Django, not CockroachDB, so please excuse any AI-generated conjectures that are incorrect.

**Describe the problem**

While running the Django test suite against a single-node, in-memory CockroachDB cluster, the `sql-exec` slow-query log recorded **616 introspection statements** exceeding the slow-query threshold over a ~75 minute run, consuming **~597 seconds (≈10 minutes) of wall-clock time** — these are the `pg_constraint` / `pg_index` queries that Django's `DatabaseIntrospection.get_constraints()` / `get_indexes()` issue once per table:

- Median latency **~940 ms**, up to **1.7 s**, *per table, per call*, regardless of table size — many of the tables involved are empty or near-empty test fixtures.

On PostgreSQL, these same statements complete in single-digit milliseconds.

**To Reproduce**

1. `cockroach start-single-node --insecure --store=type=mem,size=90%`
2. Log queries greater than 500ms: `SET CLUSTER SETTING sql.log.slow_query.latency_threshold = '500ms';`
3. Run Django's test suite: https://github.com/cockroachdb/django-cockroachdb

**Expected behavior**

The queries aren't so slow.

**Additional data / screenshots**

| Statement class | Count | Min age | Median age | Max age | Total time |
|---|---|---|---|---|---|
| `pg_index`/`pg_attribute` index-introspection SELECT | 310 | 711 ms | 1082 ms | 1718 ms | 341.0 s |
| `pg_constraint`-based constraint-introspection SELECT | 306 | 660 ms | 808 ms | 1197 ms | 256.1 s |
| **Total** | **616** | 660 ms | ~940 ms | 1718 ms | **597.1 s** |

The index/constraint introspection query was issued repeatedly for the same table across different test cases (e.g. `inspectdb_columntypes` 13 times, `inspectdb_compositepkmodel` 8 times) across 90 distinct tables, each call taking ~700–1700 ms *every time*, including for tables with zero or a handful of rows. This strongly suggests the cost is **not proportional to the target table**, but instead proportional to some cluster/database-wide metadata state (e.g. the full descriptor set) that the `pg_catalog` virtual tables (`pg_constraint`, `pg_index`, `pg_attribute`, `pg_class`) must materialize before the `WHERE cl.relname = ''` filter is applied — i.e. an O(number of descriptors in the database) cost paid on every single-table lookup, turning schema introspection of an n-table database into O(n²) total work.

Example query (age 1067 ms) — this is exactly the query pattern `django.db.backends.postgresql.introspection` issues:

```sql
SELECT indexname, array_agg(attname ORDER BY arridx), indisunique, indisprimary,
array_agg(ordering ORDER BY arridx), amname, exprdef, s2.attoptions
FROM (
SELECT c2.relname AS indexname, idx.*, attr.attname, am.amname,
CASE WHEN idx.indexprs IS NOT NULL THEN pg_get_indexdef(idx.indexrelid) END AS exprdef,
CASE am.amname WHEN 'prefix' THEN CASE (option & 1) WHEN 1 THEN 'DESC' ELSE 'ASC' END END AS ordering,
c2.reloptions AS attoptions
FROM (SELECT * FROM "".""".pg_index AS i,
ROWS FROM (unnest(i.indkey, i.indoption)) WITH ORDINALITY AS koi (key, option, arridx)) AS idx
LEFT JOIN "".""".pg_class AS c ON idx.indrelid = c.oid
LEFT JOIN "".""".pg_class AS c2 ON idx.indexrelid = c2.oid
LEFT JOIN "".""".pg_am AS am ON c2.relam = am.oid
LEFT JOIN "".""".pg_attribute AS attr ON (attr.attrelid = c.oid) AND (attr.attnum = idx.key)
WHERE (c.relname = 'composite_pk_user') AND pg_table_is_visible(c.oid)
) AS s2
GROUP BY indexname, indisunique, indisprimary, amname, exprdef, attoptions;
```

and the constraint counterpart (age 1055 ms):

```sql
SELECT c.conname,
ARRAY(SELECT attname FROM ROWS FROM (unnest(c.conkey)) WITH ORDINALITY AS cols (colid, arridx)
JOIN "".""".pg_attribute AS ca ON cols.colid = ca.attnum
WHERE ca.attrelid = c.conrelid ORDER BY cols.arridx),
c.contype,
(SELECT (fkc.relname || '.') || fka.attname
FROM "".""".pg_attribute AS fka
JOIN "".""".pg_class AS fkc ON fka.attrelid = fkc.oid
WHERE (fka.attrelid = c.confrelid) AND (fka.attnum = c.confkey[1])),
cl.reloptions
FROM "".""".pg_constraint AS c
JOIN "".""".pg_class AS cl ON c.conrelid = cl.oid
WHERE ((cl.relname = 'composite_pk_user') AND pg_table_is_visible(cl.oid)) AND (c.contype != 'n');
```

**Environment:**
- CockroachDB version v26.3.0-beta.3 (x86_64-pc-linux-gnu, built 2026/07/02 18:25:21, go1.26.4)

Jira issue: CRDB-66376

Epic CRDB-62270

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.