cockroachdb / cockroachdb/cockroach

pkg/util/log: don't panic applying log config after init log config is used

Open
#110,028 0 comments 0 reactions 0 assignees View on GitHub
A-observability-inf C-bug T-supportability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

CockroachDB has a global log configuration that's used process-wide. Once a log config has been used, that config is marked as "active". Once active, if any code attempts to apply a *new* log config, the CRDB process panics (or, if a release binary, generates a Sentry report - see https://github.com/cockroachdb/cockroach/pull/109365).

Generally, this is a strategy that makes sense. CockroachDB does not support applying new log configurations without a process restart, so any attempts to do so should crash the process.

However, there is a scenario where this doesn't make much sense.

When CockroachDB is *first initializing* - think `init()` function execution and assignment of pkg-level `var`s - a barebones log configuration is applied and used. This ensures that *some* logging facility is available during initialization, prior to the *actual*, *user-provided* log configuration being applied.

The problem here is that if this barebones log configuration is used at any time during the initialization process (for example: https://github.com/cockroachdb/cockroach/issues/109900), that barebones log config is marked "active". Then, when the node startup process finally attempts to apply the user-provided log configuration, this case is hit which causes a panic: https://github.com/cockroachdb/cockroach/blob/b2775068595438cdbab284f8f8c63fa0374cdc6e/pkg/util/log/flags.go#L95-L97

**Expected behavior**
We should tolerate a log config being applied *after* the barebones log config, even if that initial config is used.

The purpose of this panic is to avoid code from applying a log config after the user-supplied log config has been applied, which should be maintained. However, what good is having a barebones log config during initialization if it can't be used without panic?

The log package should be able to discern this barebones config from a legitimate config, so that this "active" condition isn't set until *after* the user-supplied config is applied.

Relates to: https://github.com/cockroachdb/cockroach/issues/84638

Jira issue: CRDB-31231

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.