cockroachdb / cockroachdb/cockroach

server/UA: server controller stuck in retry loop around service mode

Open
#120,201 1 comment 0 reactions 0 assignees View on GitHub
C-bug O-testcluster T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A test cluster was observed to be writing thousands of event log entries of the form:

```
Event: tenant_shared_service_start, content: { "Timestamp": 1710164069799106000, "EventType": "tenant_shared_service_start", "NodeID": 5, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164069029073000, "EventType": "tenant_shared_service_start", "NodeID": 8, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068651254500, "EventType": "tenant_shared_service_start", "NodeID": 1, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068619773700, "EventType": "tenant_shared_service_start", "NodeID": 6, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068386637800, "EventType": "tenant_shared_service_start", "NodeID": 9, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068298013000, "EventType": "tenant_shared_service_start", "NodeID": 4, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
```

Uncertain on steps to reproduce, or when exactly this started. The cluster had restored a tenant, which went straight to service mode shared, then service was stopped hours later. The tenant info was then manually edited with pb_to_json to set an unrelated field, however that update did not touch the service mode column.

Jira issue: CRDB-36542

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.