cockroachdb / cockroachdb/cockroach
changefeedccl: error decoding key during TestChangefeedNemeses
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
TestChangefeedNemeses/nemeses_options=={EnableFpValidator: false,EnableSQLSmith: true} fails under stress.
```
stepping through state reverting with unexpected error: error decoding key ‹/Table/106/110/1/0›@1733849403.305284000,0
```
More details on the changefeed statement:
```
I241210 16:49:28.231472 3384 12@util/log/event_log.go:39 ⋮ [T1,Vsystem,n1,job=1028190919862616065,change-aggregator] 512 ={"Timestamp":1733849368231469000,"EventType":"changefeed_emitted_bytes","Description":"‹CREATE CHANGEFEED FOR TABLE foo INTO 'pulsar://testfeed?region=testfeedRegion' WITH OPTIONS (diff, resolved, updated)›","SinkType":"pulsar","NumTables":1,"Resolved":"yes","InitialScan":"yes","JobId":1028190919862616065,"EmittedBytes":775,"EmittedMessages":5,"LoggingInterval":86400000000000,"Closing":true}
I241210 16:49:28.231516 3378 ccl/changefeedccl/changefeed_stmt.go:1307 ⋮ [T1,Vsystem,n1,job=CHANGEFEED id=1028190919862616065] 513 CHANGEFEED 1028190919862616065 shutting down (cause: context canceled)
W241210 16:49:28.231574 3378 ccl/changefeedccl/changefeed_stmt.go:1217 ⋮ [T1,Vsystem,n1,job=CHANGEFEED id=1028190919862616065] 514 job failed (context canceled)
```
Terminal error:
```
W241210 16:50:05.972643 121282 ccl/changefeedccl/changefeed_stmt.go:1217 ⋮ [T1,Vsystem,n1,job=CHANGEFEED id=1028190962947063809] 7351 job failed (error decoding key ‹/Table/106/110/1/0›@1733849403.305284000,0 (hex_kv: ‹0a05f2f66e8988124a0a3c000000000a261e313733333834393430323638363835353030302e303030303030303030308156017816017836017826017881260178160178160178120a08a0a3a39089dbf78718›): unknown tag 110)
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 CHANGEFEED job 1028190962947063809: stepping through state reverting with unexpected error: error decoding key ‹/Table/106/110/1/0›@1733849403.305284000,0 (hex_kv: ‹0a05f2f66e8988124a0a3c000000000a261e313733333834393430323638363835353030302e303030303030303030308156017816017836017826017881260178160178160178120a08a0a3a39089dbf78718›): unknown tag 110
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 +(1) forced error mark
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | ‹"terminal changefeed error"›
```
```
unexpected error: error decoding key ‹/Table/106/110/1/0›@1733849403.305284000,0 (hex_kv: ‹0a05f2f66e8988124a0a3c000000000a261e313733333834393430323638363835353030302e303030303030303030308156017816017836017826017881260178160178160178120a08a0a3a39089dbf78718›): unknown tag 110
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 +(1) forced error mark
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | ‹"terminal changefeed error"›
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeedbase/*changefeedbase.terminalError::
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 +Wraps: (2) attached stack trace
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + -- stack trace:
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdcevent.(*eventDecoder).DecodeKV
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | pkg/ccl/changefeedccl/cdcevent/event.go:557
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | [...repeated from below...]
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 +Wraps: (3) error decoding key ‹/Table/106/110/1/0›@1733849403.305284000,0 (hex_kv: ‹0a05f2f66e8988124a0a3c000000000a261e313733333834393430323638363835353030302e303030303030303030308156017816017836017826017881260178160178160178120a08a0a3a39089dbf78718›)
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 +Wraps: (4) attached stack trace
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + -- stack trace:
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/pkg/util/encoding.PeekLength
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | pkg/util/encoding/encoding.go:2094
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/pkg/sql/rowenc.EncDatumFromBuffer
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | pkg/sql/rowenc/encoded_datum.go:148
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/pkg/sql/rowenc.DecodeKeyValsUsingSpec
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | pkg/sql/rowenc/index_encoding.go:569
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).DecodeIndexKey
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | pkg/sql/row/fetcher.go:849
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).nextKey
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | pkg/sql/row/fetcher.go:783
E241210 16:50:05.972717 121282 jobs/registry.go:1593 ⋮ [T1,Vsystem,n1] 7352 + | github.com/cockroachdb/cockroach/p
```
Logs:
[logTestChangefeedNemeses2982758917.zip](https://github.com/user-attachments/files/18085144/logTestChangefeedNemeses2982758917.zip)
Jira issue: CRDB-45398
Contributor guide
Assessment
This issue has not been assessed yet.