cockroachdb / cockroachdb/cockroach

Sentry: version_set.go:414: log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable] (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(...

Open
#174,791 3 comments 0 reactions 0 assignees View on GitHub
A-storage C-bug O-sentry T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7715757287/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7715757287/?referrer=webhooks_plugin)

Panic Message:

```
version_set.go:414: log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble.(*versionSet).initNewDB
| external/com_github_cockroachdb_pebble/version_set.go:414
| github.com/cockroachdb/pebble.openInternal
| external/com_github_cockroachdb_pebble/open.go:464
| github.com/cockroachdb/pebble.Open
| external/com_github_cockroachdb_pebble/open.go:79
| github.com/cockroachdb/cockroach/pkg/storage.newPebble
| pkg/storage/pebble.go:1407
| github.com/cockroachdb/cockroach/pkg/storage.Open
| pkg/storage/open.go:666
| github.com/cockroachdb/cockroach/pkg/server.(*Config).CreateEngines
| pkg/server/config.go:1110
| github.com/cockroachdb/cockroach/pkg/server.NewServer
| pkg/server/server.go:319
| github.com/cockroachdb/cockroach/pkg/cli.runStart.func1
| start.go:382
| github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1.2
| start.go:826
| github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1
| start.go:886
| runtime.goexit
| src/runtime/asm_arm64.s:1447
Wraps: (2) secondary error attachment
| MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).rotate
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/rotate.go:129
| | [...repeated from below...]
| Wraps: (2) MANIFEST create failed
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/basalt/basaltfs.(*FS).Create
| | external/com_github_cockroachdb_basalt_basaltfs/fs.go:955
| | github.com/cockroachdb/cockroach/pkg/storage/fs.(*Env).Create
| | pkg/storage/fs/fs.go:404
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).rotate
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/rotate.go:124
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).runRotation
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/rotate.go:54
| | github.com/cockroachdb/pebble/internal/manifestmanager.(*Manager).CreateInitial
| | external/com_github_cockroachdb_pebble/internal/manifestmanager/manifestmanager.go:588
| | github.com/cockroachdb/pebble.(*manifestWriter).CreateInitial
| | external/com_github_cockroachdb_pebble/manifest_writer.go:217
| | github.com/cockroachdb/pebble.(*versionSet).initNewDB
| | external/com_github_cockroachdb_pebble/version_set.go:398
| | github.com/cockroachdb/pebble.openInternal
| | external/com_github_cockroachdb_pebble/open.go:464
| | github.com/cockroachdb/pebble.Open
| | external/com_github_cockroachdb_pebble/open.go:79
| | github.com/cockroachdb/cockroach/pkg/storage.newPebble
| | pkg/storage/pebble.go:1407
| | github.com/cockroachdb/cockroach/pkg/storage.Open
| | pkg/storage/open.go:666
| | github.com/cockroachdb/cockroach/pkg/server.(*Config).CreateEngines
| | pkg/server/config.go:1110
| | github.com/cockroachdb/cockroach/pkg/server.NewServer
| | pkg/server/server.go:319
| | github.com/cockroachdb/cockroach/pkg/cli.runStart.func1
| | start.go:382
| | github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1.2
| | start.go:826
| | github.com/cockroachdb/cockroach/pkg/cli.createAndStartServerAsync.func1
| | start.go:886
| | runtime.goexit
| | src/runtime/asm_arm64.s:1447
| Wraps: (4) creating object for ×
| Wraps: (5) grpc: × [code 14/Unavailable]
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *status.Error
Wraps: (3) log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: MANIFEST create failed: creating object for ×: grpc: × [code 14/Unavailable]
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
version_set.go:414: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1446-L1448](src/runtime/asm_arm64.s#L1446-L1448)
[start.go#L885-L887](start.go#L885-L887)
[start.go#L825-L827](start.go#L825-L827)
[start.go#L381-L383](start.go#L381-L383)
[pkg/server/server.go#L318-L320](pkg/server/server.go#L318-L320)
[pkg/server/config.go#L1109-L1111](pkg/server/config.go#L1109-L1111)
[pkg/storage/open.go#L665-L667](pkg/storage/open.go#L665-L667)
[pkg/storage/pebble.go#L1406-L1408](pkg/storage/pebble.go#L1406-L1408)
[external/com_github_cockroachdb_pebble/open.go#L78-L80](external/com_github_cockroachdb_pebble/open.go#L78-L80)
[external/com_github_cockroachdb_pebble/open.go#L463-L465](external/com_github_cockroachdb_pebble/open.go#L463-L465)
[external/com_github_cockroachdb_pebble/version_set.go#L413-L415](external/com_github_cockroachdb_pebble/version_set.go#L413-L415)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1447
start.go in pkg/cli.createAndStartServerAsync.func1 at line 886
start.go in pkg/cli.createAndStartServerAsync.func1.2 at line 826
start.go in pkg/cli.runStart.func1 at line 382
pkg/server/server.go in pkg/server.NewServer at line 319
pkg/server/config.go in pkg/server.(*Config).CreateEngines at line 1110
pkg/storage/open.go in pkg/storage.Open at line 666
pkg/storage/pebble.go in pkg/storage.newPebble at line 1407
external/com_github_cockroachdb_pebble/open.go in github.com/cockroachdb/pebble.Open at line 79
external/com_github_cockroachdb_pebble/open.go in github.com/cockroachdb/pebble.openInternal at line 464
external/com_github_cockroachdb_pebble/version_set.go in github.com/cockroachdb/pebble.(*versionSet).initNewDB at line 414
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.3.2 |
| Go Version | go1.26.6 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.3.3-beta.0-121382-g508a04bb95-basalt |
| Cockroach SHA | 508a04bb951c0e1158db354717633bb74dd1e741 |
| # of CPUs | 4 |
| # of Goroutines | 23 |

Jira issue: CRDB-67987

Contributor guide

Open the contributing guide

Research direction

Start at pkg/storage/pebble.go:1407 and pkg/storage/open.go:666, then follow the listed Pebble MANIFEST creation path through version_set.go:398-414 and pkg/storage/fs/fs.go:404. Use the Sentry report and its Linux arm64 and v26.3.2 context to investigate the gRPC Unavailable failure; done means the startup crash is explained and the MANIFEST creation path behaves correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.