cockroachdb / cockroachdb/cockroach
storage: panic when opts.WALFailover.Secondary.FS is nil
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
A CockroachDB node crashed due to a panic resulting from a nil pointer dereference when starting up:
```
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: + exec /usr/local/bin/crdb start --certs-dir=/home/ubuntu/.cockroach-certs --advertise-addr=172.31.30.196:26258 --advertise-sql-addr=252d1-sql.seanc.dev:26257 --attrs=node_i-0e0335631fabd2084:node_role_kv_hot --cache=26541MiB --cluster-name=252d1 --external-io-enable-non-admin-implicit-access --http-addr=:8080 --join=172.31.3.37:26258,172.31.8.27:26258,172.31.28.88:26258,172.31.40.88:26258,172.31.42.132:26258 --listen-addr=:26258 --locality=country=us,region=us-west-2,az=us-west-2a,pg=us-west-2a2 --log-config-file=/home/ubuntu/logs/crdb_log.cfg --max-offset=500ms --max-go-memory=29491MiB --max-sql-memory=8847MiB --spatial-libs=/usr/local/lib/cockroach --sql-addr=:26257 '--store=path=/mnt/data1/cockroach,attrs=vol-084d1003915faa086:type_gp3:bw_125_mibps:iops_3000:size_500_gib,size=475 GiB,provisioned-rate=bandwidth=118 MiB/s,pebble=[Options] wal_dir=/mnt/data1/wal [WAL Failover] unhealthy_operation_latency_threshold=100ms'
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: *
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * ERROR: a panic has occurred!
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * runtime error: invalid memory address or nil pointer dereference
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * (1) attached stack trace
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * -- stack trace:
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | runtime.gopanic
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | GOROOT/src/runtime/panic.go:791
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | runtime.panicmem
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | GOROOT/src/runtime/panic.go:262
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | runtime.sigpanic
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | GOROOT/src/runtime/signal_unix.go:917
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/pebble.mkdirAllAndSyncParents
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | external/com_github_cockroachdb_pebble/checkpoint.go:94
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/pebble.prepareAndOpenDirs
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | external/com_github_cockroachdb_pebble/open.go:687
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/pebble.Open
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | external/com_github_cockroachdb_pebble/open.go:105
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/cockroach/pkg/storage.newPebble
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | pkg/storage/pebble.go:940
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/cockroach/pkg/storage.Open
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | pkg/storage/open.go:557
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/cockroach/pkg/server.(*Config).CreateEngines
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | pkg/server/config.go:862
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/cockroach/pkg/server.NewServer
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | pkg/server/server.go:293
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/cockroach/pkg/cli.runStart.func1
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | start.go:364
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1.2
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | start.go:805
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | start.go:865
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | runtime.goexit
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * | src/runtime/asm_arm64.s:1223
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Jun 04 23:11:07 ip-172-31-30-196 crdb[34962]: * Error types: (1) *withstack.withStack (2) runtime.errorString
```
**To Reproduce**
```
Jun 04 22:59:34 ip-172-31-30-196 crdb[27527]: + exec /usr/local/bin/crdb start --certs-dir=/home/ubuntu/.cockroach-certs --advertise-addr=172.31.30.196:26258 --advertise-sql-addr=252d1-sql.seanc.dev:26257 --attrs=node_i-0e0335631fabd2084:node_role_kv_hot --cache=26541MiB --cluster-name=252d1 --external-io-enable-non-admin-implicit-access --http-addr=:8080 --join=172.31.3.37:26258,172.31.8.27:26258,172.31.28.88:26258,172.31.40.88:26258,172.31.42.132:26258 --listen-addr=:26258 --locality=country=us,region=us-west-2,az=us-west-2a,pg=us-west-2a2 --log-config-file=/home/ubuntu/logs/crdb_log.cfg --max-offset=500ms --max-go-memory=29491MiB --max-sql-memory=8847MiB --spatial-libs=/usr/local/lib/cockroach --sql-addr=:26257 '--store=path=/mnt/data1/cockroach,attrs=vol-084d1003915faa086:type_gp3:bw_125_mibps:iops_3000:size_500_gib,size=475 GiB,provisioned-rate=bandwidth=118 MiB/s,pebble=[Options] wal_dir=/mnt/data1/wal [WAL Failover] unhealthy_operation_latency_threshold=100ms'
```
What did you do? Describe in your own words.
Started crdb:
```
root@ip-172-31-30-196:/home/ubuntu# ll /mnt/data1/
total 32
drwxr-xr-x 5 root root 4096 Jun 4 22:21 ./
drwxr-xr-x 3 root root 4096 Jun 4 22:21 ../
drwx------ 4 ubuntu ubuntu 4096 Jun 4 23:04 cockroach/
drwx------ 2 root root 16384 Jun 4 22:21 lost+found/
drwx------ 2 ubuntu ubuntu 4096 Jun 4 22:21 wal/
root@ip-172-31-30-196:/home/ubuntu# ll /home/ubuntu/.cockroach-certs/
total 28
drwxr-x--- 2 ubuntu ubuntu 4096 Jun 4 22:21 ./
drwxr-x--- 6 ubuntu ubuntu 4096 Jun 4 22:57 ../
-rw-r--r-- 1 ubuntu ubuntu 1205 Jun 4 22:21 ca.crt
-rw-r--r-- 1 ubuntu ubuntu 1176 Jun 4 22:21 client.root.crt
-rw------- 1 ubuntu ubuntu 1675 Jun 4 22:21 client.root.key
-rw-r--r-- 1 ubuntu ubuntu 1338 Jun 4 22:21 node.crt
-rw------- 1 ubuntu ubuntu 1679 Jun 4 22:21 node.key
root@ip-172-31-30-196:/home/ubuntu# ll /home/ubuntu/logs/crdb_log.cfg
-rw-r--r-- 1 root root 2133 Jun 4 22:21 /home/ubuntu/logs/crdb_log.cfg
root@ip-172-31-30-196:/home/ubuntu# ll /usr/local/lib/cockroach/
total 12
drwxr-xr-x 3 root root 4096 Jun 4 22:02 ./
drwxr-xr-x 4 root root 4096 Jun 4 22:02 ../
lrwxrwxrwx 1 root root 43 Jun 4 22:02 libgeos.so -> /usr/local/lib/cockroach/v25.2.0/libgeos.so*
lrwxrwxrwx 1 root root 45 Jun 4 22:02 libgeos_c.so -> /usr/local/lib/cockroach/v25.2.0/libgeos_c.so*
drwxr-xr-x 2 root root 4096 Jun 4 22:02 v25.2.0/
root@ip-172-31-30-196:/home/ubuntu# ll /usr/local/lib/cockroach/v25.2.0/
total 14272
drwxr-xr-x 2 root root 4096 Jun 4 22:02 ./
drwxr-xr-x 3 root root 4096 Jun 4 22:02 ../
-rwxr-xr-x 1 root root 14039176 May 15 21:17 libgeos.so*
-rwxr-xr-x 1 root root 564616 May 15 21:17 libgeos_c.so*
root@ip-172-31-30-196:/home/ubuntu# ll /mnt/data1/cockroach/
total 20
drwx------ 4 ubuntu ubuntu 4096 Jun 4 23:06 ./
drwxr-xr-x 5 root root 4096 Jun 4 22:21 ../
-rw-r----- 1 ubuntu ubuntu 0 Jun 4 23:06 LOCK
drwxr-x--- 2 ubuntu ubuntu 4096 Jun 4 22:21 auxiliary/
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 4 23:06 cockroach-temp131803265/
-rw-r----- 1 ubuntu ubuntu 45 Jun 4 23:06 temp-dirs-record.txt
root@ip-172-31-30-196:/home/ubuntu# ll /mnt/data1/wal/
total 8
drwx------ 2 ubuntu ubuntu 4096 Jun 4 22:21 ./
drwxr-xr-x 5 root root 4096 Jun 4 22:21 ../
```
**Expected behavior**
Not panic w/ a nil pointer dereference.
```
root@ip-172-31-30-196:/home/ubuntu# crdb version
Build Tag: v25.2.0
Build Time: 2025/05/15 21:07:42
Distribution: CCL
Platform: linux arm64 (aarch64-unknown-linux-gnu)
Go Version: go1.23.7 X:nocoverageredesign
C Compiler: gcc 6.5.0
Build Commit ID: 22cec2232cf3ab2a0e5f93cd35aa1217d62a4400
Build Type: release
Enabled Assertions: false
root@ip-172-31-30-196:/home/ubuntu# uname -a
Linux ip-172-31-30-196 6.8.0-1029-aws #31-Ubuntu SMP Wed Apr 23 18:20:04 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
root@ip-172-31-30-196:/home/ubuntu# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
```
**Additional context**
I think this has to do with WAL configs. Removing the WAL options from pebble fixes the crash:
```
export COCKROACH_WAL_FAILOVER=among-stores
--- /mnt/crdb_storage_flags.txt 2025-06-04 23:29:51.685915742 +0000
+++ /mnt/crdb_storage_flags.txt2 2025-06-04 23:28:38.184288114 +0000
@@ -1 +1 @@
---store=path=/mnt/data1/cockroach,attrs=vol-084d1003915faa086:type_gp3:bw_125_mibps:iops_3000:size_500_gib,size=475 GiB,provisioned-rate=bandwidth=118 MiB/s
+--store=path=/mnt/data1/cockroach,attrs=vol-084d1003915faa086:type_gp3:bw_125_mibps:iops_3000:size_500_gib,size=475 GiB,provisioned-rate=bandwidth=118 MiB/s,pebble=[Options] wal_dir=/mnt/data1/wal [WAL Failover] unhealthy_operation_latency_threshold=100ms
```
Jira issue: CRDB-51258
Contributor guide
Assessment
This issue has not been assessed yet.