cockroachdb / cockroachdb/cockroach

sql: typeSchemaChanger.execWithRetry silently returns nil on context cancellation

Open
#166,622 1 comment 0 reactions 0 assignees View on GitHub
branch-master C-bug O-agent T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

`typeSchemaChanger.execWithRetry` (`pkg/sql/type_change.go:1399`) returns `nil` when its retry loop exits due to context cancellation, causing the type schema change job to be marked as **succeeded** even though the schema change may not have completed. This can leave enum members permanently stuck in a `READ_ONLY` transitional state.

## Root Cause

The retry loop (lines 1406-1429) has no `MaxRetries` or `MaxDuration`, so the only way it exits without an explicit `return` is when `r.Next()` returns `false` due to context cancellation. After the loop, the function unconditionally returns `nil` (line 1430).

`context.Canceled` is classified as non-permanent by `IsPermanentSchemaChangeError` (explicitly listed in `schema_changer.go:248`), so it gets retried until the context is done.

## Evidence of Intent

The analogous retry loop in `schema_changer.go:3170-3213` correctly handles this case by capturing and returning `scErr` after loop exit with the comment: *"If the context was canceled, the job registry will retry the job."*

## Impact

- Enum members can be left permanently in `READ_ONLY` state
- Users observe enum values that cannot be used in writes but remain visible in reads
- The job appears succeeded in `SHOW JOBS`
- Can occur during node drain, rolling restarts, or any scenario where the job context is canceled

## Suggested Fix

Change line 1430 from `return nil` to `return ctx.Err()`.

## Secondary Issue

The `populateIDsToRemove` closure (line 408) uses the outer `ctx` instead of its closure parameter `holder`, unlike every other closure in the function. Fix: rename `holder` to `ctx`.

_This issue was found via automated deep static analysis._

Jira issue: CRDB-62034

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.