cockroachdb / cockroachdb/cockroach

roachtest: schemachange/mixed-versions

Open
#173,241 1 comment 0 reactions 0 assignees View on GitHub
branch-master C-bug T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The `schemachange/mixed-versions` nightly failed in post-test assertions after the workload and mixed-version test body completed successfully.

- Build: [21458012](https://teamcity.cockroachdb.com/viewLog.html?buildId=21458012)
- SHA: [`8a65526f0fa3306086fa0dfa90cc24c15e8c2c82`](https://github.com/cockroachdb/cockroach/commit/8a65526f0fa3306086fa0dfa90cc24c15e8c2c82)
- Upgrade: v26.2.1 -> v26.4.0-alpha.00000000-dev
- Artifacts: [artifacts.zip](https://teamcity.cockroachdb.com/repository/download/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateStagingBazel/21458012:id/schemachange/mixed-versions/run_1/artifacts.zip), [debug.zip](https://teamcity.cockroachdb.com/repository/download/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateStagingBazel/21458012:id/schemachange/mixed-versions/run_1/debug.zip)

The final post-test `INSPECT` failed with:

```
INSPECT found 1 consistency errors:
Job 1200089706597711873:
- schemachange.public.table_w3_35: unsafe_split_key
range r202 start_key=/Table/133/12/... (table_id=133, index_id=12): split key is mid-column-family
```

## Triage

This is a genuine unsafe SQL-row split, not a GCE or replica-health failure. KV consistency passed, all nodes remained healthy, and the mixed-version test body completed before the post-test assertion failed.

The causal sequence is:

1. At `20:46:20`, the schemachange workload attempted:

```sql
CREATE UNIQUE INDEX index_w4_81_w4_134
ON public.table_w4_81
(col81_w4_85 DESC, col81_w4_86 DESC, col81_w4_83,
col81_w4_88 DESC, col81_w4_87 ASC)
STORING (col81_w4_82)
```

2. During the declarative schema changer pre-commit phase, `index_split_scatter.go` generated 33 statistics-derived split points for each of indexes 11 and 12, selected 16 for each, and issued raw `AdminSplitRequest`s. Range `r202` was manually split at the key later reported by `INSPECT`.

3. At `20:46:24`, the SQL transaction failed to commit with `SQLSTATE 40001`. The index and schema-change job were not committed, but the non-transactional, approximately one-hour sticky range splits remained.

4. At `20:50:30`, a successful `TRUNCATE TABLE ... table_w4_81 ...` recreated the table's indexes. Because the aborted transaction had not advanced the descriptor's index-ID allocation, the truncate reused IDs 11 and 12. Index ID 12 now described `table_w4_81_col81_w4_82_idx`, whose key schema is unrelated to the aborted five-column index.

5. The surviving `/Table/133/12/...` boundary was consequently interpreted under the new index schema and became a mid-column-family split. `INSPECT` detected it at `20:51:53`.

The stats/copy batch path still sends split points directly as raw `AdminSplitRequest`s: [`pkg/sql/index_split_scatter.go`](https://github.com/cockroachdb/cockroach/blob/8a65526f0fa3306086fa0dfa90cc24c15e8c2c82/pkg/sql/index_split_scatter.go#L397-L432). The split was valid for the transient index schema when created; the failure is that it survived the transaction rollback and its index ID was subsequently reused.

Mixed-version execution exposed the sequence, but does not appear causal. The split was created by a v26.2.1 node and the later truncate ran on a current-version node; the same rollback-and-ID-reuse lifecycle can occur without an upgrade.

## Diagnostic wrinkle

The reported object name, `table_w3_35` (descriptor 143), is misleading. The bad key belongs to descriptor 133, `table_w4_81`. Range `r202` starts inside table 133 and ends at `/Table/149/3`, so it overlaps table 143's inspection span. The split-key check validates the start key of every overlapping range and then attributes an error to the table whose check is currently running, rather than the table ID decoded from the offending key.

## Related issues

- #43094 tracks the underlying invariant that splits may divide SQL rows.
- #112702 discusses the encoding and read-corruption implications of unsafe split keys.
- #172140 added post-backfill split validation, which detects this bug class but does not prevent sticky splits created for uncommitted index IDs.

I did not find an issue tracking this exact manifestation: statistics-derived pre-commit split, transaction rollback, followed by `TRUNCATE` reusing the index ID.

The likely fix area is the lifecycle of pre-commit index-backfill splits: do not leave sticky splits for uncommitted index IDs, or remove/translate them when the schema-change transaction rolls back before those IDs can be reused.

Jira issue: CRDB-66557

Epic CRDB-65516

Contributor guide

Open the contributing guide

Research direction

Start in pkg/sql/index_split_scatter.go around the stats/copy batch path that sends raw AdminSplitRequests. Trace how pre-commit index-backfill splits survive a failed schema-change transaction and how TRUNCATE can reuse the index ID; compare the behavior with issues #43094, #112702, and #172140. Done means uncommitted index splits are removed or otherwise cannot become unsafe after rollback and ID reuse.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.