cockroachdb / cockroachdb/cockroach
Sentry: version_set.go:414: log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable] (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7715757287/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7715757287/?referrer=webhooks_plugin)
Panic Message:
```
version_set.go:414: log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble.(*versionSet).initNewDB
| external/com_github_cockroachdb_pebble/version_set.go:414
| github.com/cockroachdb/pebble.openInternal
| external/com_github_cockroachdb_pebble/open.go:464
| github.com/cockroachdb/pebble.Open
| external/com_github_cockroachdb_pebble/open.go:79
| github.com/cockroachdb/cockroach/pkg/storage.newPebble
| pkg/storage/pebble.go:1407
| github.com/cockroachdb/cockroach/pkg/storage.Open
| pkg/storage/open.go:666
| github.com/cockroachdb/cockroach/pkg/server.(*Config).CreateEngines
| pkg/server/config.go:1110
| github.com/cockroachdb/cockroach/pkg/server.NewServer
| pkg/server/server.go:319
| github.com/cockroachdb/cockroach/pkg/cli.runStart.func1
| start.go:382
| github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1.2
| start.go:826
| github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1
| start.go:886
| runtime.goexit
| src/runtime/asm_arm64.s:1447
Wraps: (2) secondary error attachment
| MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).rotate
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/rotate.go:129
| | [...repeated from below...]
| Wraps: (2) MANIFEST create failed
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/basalt/basaltfs.(*FS).Create
| | external/com_github_cockroachdb_basalt_basaltfs/fs.go:955
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Create
| | pkg/storage/fs/fs.go:404
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).rotate
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/rotate.go:124
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).runRotation
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/rotate.go:54
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).CreateInitial
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/manifestmanager.go:588
| | github.com/cockroachdb/pebble.(*manifestWriter).CreateInitial
| | external/com_github_cockroachdb_pebble/manifest_writer.go:217
| | github.com/cockroachdb/pebble.(*versionSet).initNewDB
| | external/com_github_cockroachdb_pebble/version_set.go:398
| | github.com/cockroachdb/pebble.openInternal
| | external/com_github_cockroachdb_pebble/open.go:464
| | github.com/cockroachdb/pebble.Open
| | external/com_github_cockroachdb_pebble/open.go:79
| | github.com/cockroachdb/cockroach/pkg/storage.newPebble
| | pkg/storage/pebble.go:1407
| | github.com/cockroachdb/cockroach/pkg/storage.Open
| | pkg/storage/open.go:666
| | github.com/cockroachdb/cockroach/pkg/server.(*Config).CreateEngines
| | pkg/server/config.go:1110
| | github.com/cockroachdb/cockroach/pkg/server.NewServer
| | pkg/server/server.go:319
| | github.com/cockroachdb/cockroach/pkg/cli.runStart.func1
| | start.go:382
| | github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1.2
| | start.go:826
| | github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1
| | start.go:886
| | runtime.goexit
| | src/runtime/asm_arm64.s:1447
| Wraps: (4) creating object for ×
| Wraps: (5) grpc: × [code 14/Unavailable]
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *status.Error
Wraps: (3) log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
version_set.go:414: *withstack.withStack (top exception)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_arm64.s#L1446-L1448](src/runtime/asm_arm64.s#L1446-L1448)
[start.go#L885-L887](start.go#L885-L887)
[start.go#L825-L827](start.go#L825-L827)
[start.go#L381-L383](start.go#L381-L383)
[pkg/server/server.go#L318-L320](pkg/server/server.go#L318-L320)
[pkg/server/config.go#L1109-L1111](pkg/server/config.go#L1109-L1111)
[pkg/storage/open.go#L665-L667](pkg/storage/open.go#L665-L667)
[pkg/storage/pebble.go#L1406-L1408](pkg/storage/pebble.go#L1406-L1408)
[external/com_github_cockroachdb_pebble/open.go#L78-L80](external/com_github_cockroachdb_pebble/open.go#L78-L80)
[external/com_github_cockroachdb_pebble/open.go#L463-L465](external/com_github_cockroachdb_pebble/open.go#L463-L465)
[external/com_github_cockroachdb_pebble/version_set.go#L413-L415](external/com_github_cockroachdb_pebble/version_set.go#L413-L415)
```
src/runtime/asm_arm64.s in runtime.goexit at line 1447
start.go in pkg/cli.createAndStartServerAsync.func1 at line 886
start.go in pkg/cli.createAndStartServerAsync.func1.2 at line 826
start.go in pkg/cli.runStart.func1 at line 382
pkg/server/server.go in pkg/server.NewServer at line 319
pkg/server/config.go in pkg/server.(*Config).CreateEngines at line 1110
pkg/storage/open.go in pkg/storage.Open at line 666
pkg/storage/pebble.go in pkg/storage.newPebble at line 1407
external/com_github_cockroachdb_pebble/open.go in github.com/cockroachdb/pebble.Open at line 79
external/com_github_cockroachdb_pebble/open.go in github.com/cockroachdb/pebble.openInternal at line 464
external/com_github_cockroachdb_pebble/version_set.go in github.com/cockroachdb/pebble.(*versionSet).initNewDB at line 414
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.3.2 |
| Go Version | go1.26.6 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.3.3-beta.0-121382-g508a04bb95-basalt |
| Cockroach SHA | 508a04bb951c0e1158db354717633bb74dd1e741 |
| # of CPUs | 4 |
| # of Goroutines | 23 |
Jira issue: CRDB-67987
Contributor guide
Research direction
Start at pkg/storage/pebble.go:1407 and pkg/storage/open.go:666, then follow the listed Pebble MANIFEST creation path through version_set.go:398-414 and pkg/storage/fs/fs.go:404. Use the Sentry report and its Linux arm64 and v26.3.2 context to investigate the gRPC Unavailable failure; done means the startup crash is explained and the MANIFEST creation path behaves correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100