cockroachdb / cockroachdb/cockroach

Sentry: errors.go:439: log.Dev.Fatal: transaction unexpectedly finalized in (*kvpb.ExclusionViolationError): write exclusion on key /Table/231/1/×/×/× expected since 1780801244.330247564,1 but foun...

Open
#171,482 1 comment 0 reactions 0 assignees View on GitHub
branch-release-26.2 C-bug O-sentry T-kv target-release-25.4.17 target-release-26.1.9 target-release-26.2.7 v25.4.13 v25.4.15 v26.1.7 v26.2.4 v26.2.6 v26.3.0-prerelease X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7533752568/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7533752568/?referrer=webhooks_plugin)

Panic Message:

```
errors.go:439: log.Dev.Fatal: transaction unexpectedly finalized in (*kvpb.ExclusionViolationError): write exclusion on key /Table/231/1/×/×/× expected since 1780801244.330247564,1 but found write at 1780801244.816938561,0: "sql txn" meta={id=935e2dd6 key=/Table/231/1/×/×/× iso=Serializable pri=0.00169505 epo=0 ts=1780801244.330247564,1 min=1780801235.417818362,0 seq=0} lock=true stat=ABORTED rts=1780801244.330247564,1 gul=1780801235.917818362,0 obs={n2@1780801244.816861031,0 n3@1780801235.417818362,0}
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*Error).checkTxnStatusValid
| pkg/kv/kvpb/errors.go:439
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*Error).UpdateTxn
| pkg/kv/kvpb/errors.go:421
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*Error).SetTxn
| pkg/kv/kvpb/errors.go:407
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.NewErrorWithTxn
| pkg/kv/kvpb/errors.go:182
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| pkg/kv/kvserver/replica_evaluate.go:446
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
| pkg/kv/kvserver/replica_write.go:766
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
| pkg/kv/kvserver/replica_write.go:731
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
| pkg/kv/kvserver/replica_write.go:494
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
| pkg/kv/kvserver/replica_proposal.go:999
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
| pkg/kv/kvserver/replica_proposal.go:1095
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
| pkg/kv/kvserver/replica_raft.go:131
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
| pkg/kv/kvserver/replica_write.go:192
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:542
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:201
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:197
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:192
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1746
| github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
| pkg/server/node.go:1939
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchStreamImpl
| pkg/server/node.go:1986
| github.com/cockroachdb/cockroach/pkg/server.(*Node).BatchStream
| pkg/server/node.go:1965
| github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_BatchStream_Handler
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:11778
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
| pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:107
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| pkg/rpc/context.go:191
| google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
| pkg/rpc/auth.go:192
| google.golang.org/grpc.getChainStreamHandler.func1
| external/org_golang_google_grpc/server.go:1504
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| pkg/rpc/context.go:148
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| pkg/util/stop/stopper.go:358
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| pkg/rpc/context.go:147
| google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
| external/org_golang_google_grpc/server.go:1495
| google.golang.org/grpc.(*Server).processStreamingRPC
| external/org_golang_google_grpc/server.go:1659
Wraps: (2) secondary error attachment
| write exclusion on key /Table/231/1/×/×/× expected since 1780801244.330247564,1 but found write at 1780801244.816938561,0
| (1) write exclusion on key /Table/231/1/×/×/× expected since 1780801244.330247564,1 but found write at 1780801244.816938561,0
| Error types: (1) *kvpb.ExclusionViolationError
Wraps: (3) log.Dev.Fatal: transaction unexpectedly finalized in (*kvpb.ExclusionViolationError): write exclusion on key /Table/231/1/×/×/× expected since 1780801244.330247564,1 but found write at 1780801244.816938561,0: "sql txn" meta={id=935e2dd6 key=/Table/231/1/×/×/× iso=Serializable pri=0.00169505 epo=0 ts=1780801244.330247564,1 min=1780801235.417818362,0 seq=0} lock=true stat=ABORTED rts=1780801244.330247564,1 gul=1780801235.917818362,0 obs={n2@1780801244.816861031,0 n3@1780801235.417818362,0}
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: transaction unexpectedly finalized in (*kvpb.ExclusionViolationError): write exclusion on key /Table/231/1/×/×/× expected since 1780801244.330247564,1 but found write at 1780801244.816938561,0: "sql txn" meta={id=935e2dd6 key=/Table/231/1/×/×/× iso=Serializable pri=0.00169505 epo=0 ts=1780801244.330247564,1 min=1780801235.417818362,0 seq=0} lock=true stat=ABORTED rts=1780801244.330247564,1 gul=1780801235.917818362,0 obs={n2@1780801244.816861031,0 n3@1780801235.417818362,0}
*secondary.withSecondaryError
errors.go:439: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[external/org_golang_google_grpc/server.go#L1658-L1660](external/org_golang_google_grpc/server.go#L1658-L1660)
[external/org_golang_google_grpc/server.go#L1494-L1496](external/org_golang_google_grpc/server.go#L1494-L1496)
[pkg/rpc/context.go#L146-L148](pkg/rpc/context.go#L146-L148)
[pkg/util/stop/stopper.go#L357-L359](pkg/util/stop/stopper.go#L357-L359)
[pkg/rpc/context.go#L147-L149](pkg/rpc/context.go#L147-L149)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/rpc/auth.go#L191-L193](pkg/rpc/auth.go#L191-L193)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/rpc/context.go#L190-L192](pkg/rpc/context.go#L190-L192)
[external/org_golang_google_grpc/server.go#L1503-L1505](external/org_golang_google_grpc/server.go#L1503-L1505)
[pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L106-L108](pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L106-L108)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/b398a6e1fde74a915daf8e9841303c8ee39c2abe/pkg/kv/kvpb/api.pb.go#L11777-L11779](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/b398a6e1fde74a915daf8e9841303c8ee39c2abe/pkg/kv/kvpb/api.pb.go#L11777-L11779)
[pkg/server/node.go#L1964-L1966](pkg/server/node.go#L1964-L1966)
[pkg/server/node.go#L1985-L1987](pkg/server/node.go#L1985-L1987)
[pkg/server/node.go#L1938-L1940](pkg/server/node.go#L1938-L1940)
[pkg/server/node.go#L1745-L1747](pkg/server/node.go#L1745-L1747)
[pkg/kv/kvserver/stores.go#L191-L193](pkg/kv/kvserver/stores.go#L191-L193)
[pkg/kv/kvserver/store_send.go#L196-L198](pkg/kv/kvserver/store_send.go#L196-L198)
[pkg/kv/kvserver/replica_send.go#L200-L202](pkg/kv/kvserver/replica_send.go#L200-L202)
[pkg/kv/kvserver/replica_send.go#L541-L543](pkg/kv/kvserver/replica_send.go#L541-L543)
[pkg/kv/kvserver/replica_write.go#L191-L193](pkg/kv/kvserver/replica_write.go#L191-L193)
[pkg/kv/kvserver/replica_raft.go#L130-L132](pkg/kv/kvserver/replica_raft.go#L130-L132)
[pkg/kv/kvserver/replica_proposal.go#L1094-L1096](pkg/kv/kvserver/replica_proposal.go#L1094-L1096)
[pkg/kv/kvserver/replica_proposal.go#L998-L1000](pkg/kv/kvserver/replica_proposal.go#L998-L1000)
[pkg/kv/kvserver/replica_write.go#L493-L495](pkg/kv/kvserver/replica_write.go#L493-L495)
[pkg/kv/kvserver/replica_write.go#L730-L732](pkg/kv/kvserver/replica_write.go#L730-L732)
[pkg/kv/kvserver/replica_write.go#L765-L767](pkg/kv/kvserver/replica_write.go#L765-L767)
[pkg/kv/kvserver/replica_evaluate.go#L445-L447](pkg/kv/kvserver/replica_evaluate.go#L445-L447)
[pkg/kv/kvpb/errors.go#L181-L183](pkg/kv/kvpb/errors.go#L181-L183)
[pkg/kv/kvpb/errors.go#L406-L408](pkg/kv/kvpb/errors.go#L406-L408)
[pkg/kv/kvpb/errors.go#L420-L422](pkg/kv/kvpb/errors.go#L420-L422)
[pkg/kv/kvpb/errors.go#L438-L440](pkg/kv/kvpb/errors.go#L438-L440)

```
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1659
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1495
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 147
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 358
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 148
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/rpc/auth.go in pkg/rpc.kvAuth.streamInterceptor at line 192
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1 at line 1504
pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 191
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 107
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb._Internal_BatchStream_Handler at line 11778
pkg/server/node.go in pkg/server.(*Node).BatchStream at line 1965
pkg/server/node.go in pkg/server.(*Node).batchStreamImpl at line 1986
pkg/server/node.go in pkg/server.(*Node).Batch at line 1939
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1746
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 192
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 197
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 201
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 542
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 192
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 131
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 1095
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 999
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 494
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 731
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 766
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 446
pkg/kv/kvpb/errors.go in pkg/kv/kvpb.NewErrorWithTxn at line 182
pkg/kv/kvpb/errors.go in pkg/kv/kvpb.(*Error).SetTxn at line 407
pkg/kv/kvpb/errors.go in pkg/kv/kvpb.(*Error).UpdateTxn at line 421
pkg/kv/kvpb/errors.go in pkg/kv/kvpb.(*Error).checkTxnStatusValid at line 439
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.2.1 |
| Go Version | go1.25.5 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v26.2.1 |
| Cockroach SHA | b398a6e1fde74a915daf8e9841303c8ee39c2abe |
| # of CPUs | 8 |
| # of Goroutines | 671 |

Jira issue: CRDB-64600

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.