cockroachdb / cockroachdb/cockroach

demo: panic with workload init

Open
#94,654 2 comments 0 reactions 0 assignees View on GitHub
A-demo C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Please describe the issue you observed, and any steps we can take to reproduce it:

**To Reproduce**

```bash
cockroach demo --nodes 9 --with-load

\demo add region=us-central1,az=a
\demo add region=us-central1,az=b
\demo add region=us-central1,az=c

\demo shutdown 7
\demo shutdown 8
\demo shutdown 9

# capture a node to connect to
\demo ls
```

in another window

```bash
cockroach workload init tpcc --db movr "postgresql://demo:demo55320@127.0.0.1:26268/movr?sslmode=require&sslrootcert=%2Fvar%2Ffolders%2F1c%2Fj4z3c4gn5gx3mftgpcmfcrsc0000gq%2FT%2Fdemo4578662%2Fca.crt"
```

init will fail with the following message

```bash
Error: importing fixture: importing table history: pq: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101533696 currently allocated, 134217728 bytes in budget
```

now attempt to exit out of the original cockroach demo terminal

```bash
demo@127.0.0.1:26257/movr> \q
*
* ERROR: a panic has occurred!
* pebble: closed
* (1) attached stack trace
* -- stack trace:
* | runtime.gopanic
* | GOROOT/src/runtime/panic.go:884
* | github.com/cockroachdb/pebble.(*DB).Apply
* | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:702
* | github.com/cockroachdb/pebble.(*Batch).Commit
* | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1049
* | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit
* | github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:545
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:983
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:664
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:641
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker
* | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:308
* | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
* | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489
* Wraps: (2) attached stack trace
* -- stack trace:
* | github.com/cockroachdb/pebble.(*DB).Close
* | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:1239
* | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).Close
* | github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1068
* | github.com/cockroachdb/cockroach/pkg/server.(*stickyInMemEnginesRegistryImpl).deleteEngine
* | github.com/cockroachdb/cockroach/pkg/server/sticky_engine.go:186
* | github.com/cockroachdb/cockroach/pkg/server.(*stickyInMemEnginesRegistryImpl).CloseAllStickyInMemEngines
* | github.com/cockroachdb/cockroach/pkg/server/sticky_engine.go:176
* | github.com/cockroachdb/cockroach/pkg/cli/democluster.(*transientCluster).Start.func3.1
* | github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster.go:286
* | github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close
* | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107
* | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
* | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558
* | github.com/cockroachdb/cockroach/pkg/cli/democluster.(*transientCluster).Close
* | github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster.go:811
* | github.com/cockroachdb/cockroach/pkg/cli.runDemoInternal
* | github.com/cockroachdb/cockroach/pkg/cli/demo.go:372
* | github.com/cockroachdb/cockroach/pkg/cli.runDemo
* | github.com/cockroachdb/cockroach/pkg/cli/demo.go:210
* | github.com/cockroachdb/cockroach/pkg/cli.init.5.func1
* | github.com/cockroachdb/cockroach/pkg/cli/demo.go:59
* | github.com/cockroachdb/cockroach/pkg/cli/clierrorplus.MaybeDecorateError.func1
* | github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/decorate_error.go:71
* | github.com/spf13/cobra.(*Command).execute
* | github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:856
* | github.com/spf13/cobra.(*Command).ExecuteC
* | github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974
* | github.com/spf13/cobra.(*Command).Execute
* | github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
* | github.com/cockroachdb/cockroach/pkg/cli.Run
* | github.com/cockroachdb/cockroach/pkg/cli/cli.go:300
* | github.com/cockroachdb/cockroach/pkg/cli.doMain
* | github.com/cockroachdb/cockroach/pkg/cli/cli.go:143
* | github.com/cockroachdb/cockroach/pkg/cli.Main
* | github.com/cockroachdb/cockroach/pkg/cli/cli.go:68
* | main.main
* | main/pkg/cmd/cockroach/main.go:26
* | [...repeated from below...]
* Wraps: (3) attached stack trace
* -- stack trace:
* | github.com/cockroachdb/pebble.init
* | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:44
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6340
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6317
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6317
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6317
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6317
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6317
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6317
* | runtime.doInit
* | GOROOT/src/runtime/proc.go:6317
* | runtime.main
* | GOROOT/src/runtime/proc.go:233
* | runtime.goexit
* | src/runtime/asm_arm64.s:1172
* Wraps: (4) pebble: closed
* Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *withstack.withStack (4) *errutil.leafError
*
*
* ERROR: Queued as error 8638dbe8c3ff487892d2381217d33e02
*
panic: pebble: closed [recovered]
panic: pebble: closed

goroutine 46696 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x140075b97a0, {0x1098d1060, 0x14011ae4a80})
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:248 +0xa0
panic({0x109495c60, 0x1402bccafa8})
GOROOT/src/runtime/panic.go:884 +0x204
github.com/cockroachdb/pebble.(*DB).Apply(0x140143b69b8?, 0x105ab08dc?, 0x140143b69b8?)
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:702 +0x378
github.com/cockroachdb/pebble.(*Batch).Commit(...)
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1049
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit(0x14034a6d000?, 0x60?)
github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:545 +0x4c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:983 +0x10a8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:664 +0x138
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x14018616000, 0x14018711740?)
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:641 +0x88
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0x14001ba43c0, {0x1098d1060, 0x14011ae4a80})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:308 +0x258
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489 +0x12c
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x358
```

**Expected behavior**

the system should prevent from getting in this state or fail gracefully.

**Environment:**
- CockroachDB version 22.2.1
- Server OS: OSX 12.6
- Client app: cockroach demo

**Additional context**

minor impact

Jira issue: CRDB-23067

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.