cockroachdb / cockroachdb/cockroach
server/UA: server controller stuck in retry loop around service mode
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A test cluster was observed to be writing thousands of event log entries of the form:
```
Event: tenant_shared_service_start, content: { "Timestamp": 1710164069799106000, "EventType": "tenant_shared_service_start", "NodeID": 5, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164069029073000, "EventType": "tenant_shared_service_start", "NodeID": 8, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068651254500, "EventType": "tenant_shared_service_start", "NodeID": 1, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068619773700, "EventType": "tenant_shared_service_start", "NodeID": 6, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068386637800, "EventType": "tenant_shared_service_start", "NodeID": 9, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
Event: tenant_shared_service_start, content: { "Timestamp": 1710164068298013000, "EventType": "tenant_shared_service_start", "NodeID": 4, "TenantName": "app", "ErrorText": "while starting server: service check failed: expected service mode shared, record says none" } Mar 11, 2024 at 13:34 UTC
```
Uncertain on steps to reproduce, or when exactly this started. The cluster had restored a tenant, which went straight to service mode shared, then service was stopped hours later. The tenant info was then manually edited with pb_to_json to set an unrelated field, however that update did not touch the service mode column.
Jira issue: CRDB-36542
Contributor guide
Assessment
This issue has not been assessed yet.