[BUG] Coordinator pod on Helm deployment fails when ReplicaSet or StatefulSet are present via Helm
- Dominant language
- C++
- Stars
- 3.6k
- Forks
- 468
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
**Describe the bug**
When deploying Graphscope with Helm sometimes I get a failure when coordinator is starting up, can reproduce multiple times.
Concrete message:
```
kubernetes.client.exceptions.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Audit-Id': '736263cd-0bc2-4de6-970e-849608c02d8b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e749963a-6c93-4734-ac08-fec4051f6564', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fcaae050-bc69-4fdd-a56b-6fb27f4774b8', 'Date': 'Fri, 19 Apr 2024 16:13:22 GMT', 'Content-Length': '254'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"statefulsets.apps \"gs-engine-graphscope\" already exists","reason":"AlreadyExists","details":{"name":"gs-engine-graphscope","group":"apps","kind":"statefulsets"},"code":409}
```
Looks to me like it is trying to control the resources controlled by helm.
**To Reproduce**
Steps to reproduce the behavior:
1. Deploy via helm:
```
helm upgrade --install graphscope graphscope/graphscope -f values.yaml --wait
```
2. Look at restarting container from deployment coordinator-graphscope
**Expected behavior**
Coordinator consistently is able to start up.
**Environment (please complete the following information):**
- GraphScope version: 2.7.0
- OS: Linux/Kubernetes
- Version: Ubuntu 20.04
- Kubernetes Version 1.28.3
**Additional context**
Full logs below:
```
2024-04-19 16:13:22,047 [INFO][coordinator:89]: Start server with args
coordinator:
deployment_name: coordinator-graphscope
endpoint: null
monitor: false
monitor_port: 9090
node_selector: null
operator_mode: false
resource:
limits: null
requests:
cpu: 0.5
memory: 512Mi
service_port: 59001
hosts_launcher:
dataset_download_retries: 3
etcd:
endpoint: null
listening_client_port: 2379
listening_peer_port: 2380
replicas: 1
hosts:
- localhost
kubernetes_launcher:
config_file: null
dataset:
enable: true
proxy: bnVsbA==
delete_namespace: false
deployment_mode: eager
engine:
enable_gae: false
enable_gae_java: false
enable_gie: false
enable_gle: false
enabled_engines: analytical,interactive
gae_resource:
limits: null
requests:
cpu: '1'
memory: 1Gi
gie_executor_resource:
limits: null
requests:
cpu: '1'
memory: 1Gi
gie_frontend_resource:
limits: null
requests:
cpu: '0.5'
memory: 1Gi
gle_resource:
limits: null
requests:
cpu: '0.2'
memory: 0.2Gi
node_selector: null
preemptive: true
image:
pull_policy: IfNotPresent
pull_secrets: []
registry: registry.cn-hongkong.aliyuncs.com
repository: graphscope
tag: 0.27.0
mars:
enable: false
scheduler_resource:
limits: null
requests:
cpu: 0.2
memory: 4Mi
worker_resource:
limits: null
requests:
cpu: 0.2
memory: 4Mi
namespace: default
service_type: NodePort
volumes: null
waiting_for_delete: false
launcher_type: k8s
log_level: info
operator_launcher:
gae_endpoint: ''
hosts: []
namespace: default
session:
dangling_timeout_seconds: -1
default_local_num_workers: 1
execution_mode: eager
instance_id: graphscope
num_workers: 1
reconnect: false
retry_time_seconds: 1
timeout_seconds: 1200
show_log: false
solution: GraphScope One
vineyard:
deployment_name: null
image: vineyardcloudnative/vineyardd:latest
resource:
limits:
cpu: '0.5'
memory: 512Mi
requests:
cpu: '0.5'
memory: 512Mi
rpc_port: 9600
socket: null
2024-04-19 16:13:22,047 [INFO][launcher:44]: Failed to resolve the openmpi path, moving towards the system-wide one
2024-04-19 16:13:22,060 [INFO][kubernetes_launcher:816]: Creating engine pods...
2024-04-19 16:13:23,321 [ERROR][kubernetes_launcher:1216]: Error when launching GraphScope on kubernetes cluster
2024-04-19 16:13:23,321 [ERROR][kubernetes_launcher:1216]: Error when launching GraphScope on kubernetes cluster
Traceback (most recent call last):
File "/home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/kubernetes_launcher.py", line 1210, in start
self._create_services()
File "/home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/kubernetes_launcher.py", line 871, in _create_services
self._create_engine_stateful_set()
File "/home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/kubernetes_launcher.py", line 827, in _create_engine_stateful_set
Traceback (most recent call last):
response = self._apps_api.create_namespaced_stateful_set(
File "/home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/kubernetes_launcher.py", line 1210, in start
self._create_services()
File "/home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/kubernetes_launcher.py", line 871, in _create_services
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api/apps_v1_api.py", line 639, in create_namespaced_stateful_set
return self.create_namespaced_stateful_set_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api/apps_v1_api.py", line 738, in create_namespaced_stateful_set_with_http_info
return self.api_client.call_api(
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
self._create_engine_stateful_set()
File "/home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/kubernetes_launcher.py", line 827, in _create_engine_stateful_set
response = self._apps_api.create_namespaced_stateful_set(
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api/apps_v1_api.py", line 639, in create_namespaced_stateful_set
return self.create_namespaced_stateful_set_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api/apps_v1_api.py", line 738, in create_namespaced_stateful_set_with_http_info
return self.api_client.call_api(
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 391, in request
return self.__call_api(resource_path, method,
return self.rest_client.POST(url,
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 279, in POST
return self.request("POST", url,
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Audit-Id': '736263cd-0bc2-4de6-970e-849608c02d8b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e749963a-6c93-4734-ac08-fec4051f6564', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fcaae050-bc69-4fdd-a56b-6fb27f4774b8', 'Date': 'Fri, 19 Apr 2024 16:13:22 GMT', 'Content-Length': '254'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"statefulsets.apps \"gs-engine-graphscope\" already exists","reason":"AlreadyExists","details":{"name":"gs-engine-graphscope","group":"apps","kind":"statefulsets"},"code":409}
response_data = self.request(
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 391, in request
return self.rest_client.POST(url,
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 279, in POST
return self.request("POST", url,
File "/home/graphscope/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Audit-Id': '736263cd-0bc2-4de6-970e-849608c02d8b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e749963a-6c93-4734-ac08-fec4051f6564', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fcaae050-bc69-4fdd-a56b-6fb27f4774b8', 'Date': 'Fri, 19 Apr 2024 16:13:22 GMT', 'Content-Length': '254'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"statefulsets.apps \"gs-engine-graphscope\" already exists","reason":"AlreadyExists","details":{"name":"gs-engine-graphscope","group":"apps","kind":"statefulsets"},"code":409}
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/lib/python3.10/runpy.py:196 in _run_module_as_main │
│ │
│ 193 │ main_globals = sys.modules["__main__"].__dict__ │
│ 194 │ if alter_argv: │
│ 195 │ │ sys.argv[0] = mod_spec.origin │
│ ❱ 196 │ return _run_code(code, main_globals, None, │
│ 197 │ │ │ │ │ "__main__", mod_spec) │
│ 198 │
│ 199 def run_module(mod_name, init_globals=None, │
│ │
│ /usr/lib/python3.10/runpy.py:86 in _run_code │
│ │
│ 83 │ │ │ │ │ __loader__ = loader, │
│ 84 │ │ │ │ │ __package__ = pkg_name, │
│ 85 │ │ │ │ │ __spec__ = mod_spec) │
│ ❱ 86 │ exec(code, run_globals) │
│ 87 │ return run_globals │
│ 88 │
│ 89 def _run_module_code(code, init_globals=None, │
│ │
│ /home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/__main__. │
│ py:3 in │
│ │
│ 1 from gscoordinator.coordinator import launch_graphscope │
│ 2 │
│ ❱ 3 launch_graphscope() │
│ 4 │
│ │
│ /home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/coordinat │
│ or.py:91 in launch_graphscope │
│ │
│ 88 │ config_logging(config.log_level) │
│ 89 │ logger.info("Start server with args \n%s", config.dumps_yaml()) │
│ 90 │ │
│ ❱ 91 │ servicer = get_servicer(config) │
2024-04-19 16:13:23,399 [INFO][service:518]: Clean up resources, cleanup_instance: True, is_dangling: False
│ 92 │ start_server(servicer, config) │
│ 93 │
│ 94 │
│ │
│ /home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/coordinat │
│ or.py:121 in get_servicer │
│ │
│ 118 │ │ │ f"Expect {service_initializers.keys()} of solution paramet │
│ 119 │ │ ) │
│ 120 │ │
│ ❱ 121 │ return initializer(config) │
│ 122 │
│ 123 │
│ 124 def start_server( │
│ │
│ /home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/servicer/ │
│ graphscope_one/service.py:591 in init_graphscope_one_service_servicer │
│ │
│ 588 │ if launcher is None: │
│ 589 │ │ raise RuntimeError(f"Expect {type2launcher.keys()} of launcher │
│ 590 │ │
│ ❱ 591 │ return GraphScopeOneServiceServicer( │
│ 592 │ │ launcher=launcher(config), │
│ 593 │ │ dangling_timeout_seconds=config.session.dangling_timeout_secon │
│ 594 │ │ log_level=config.log_level, │
│ │
│ /home/graphscope/.local/lib/python3.10/site-packages/gscoordinator/servicer/ │
│ graphscope_one/service.py:117 in __init__ │
│ │
│ 114 │ │ self._launcher = launcher │
│ 115 │ │ self._launcher.set_session_workspace(self._session_id) │
│ 116 │ │ if not self._launcher.start(): │
│ ❱ 117 │ │ │ raise RuntimeError("Coordinator launching instance failed. │
│ 118 │ │ │
│ 119 │ │ self._operation_executor: OperationExecutor = OperationExecuto │
│ 120 │ │ │ self._session_id, self._launcher, self._object_manager │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Coordinator launching instance failed.
2024-04-19 16:13:23,434 [INFO][service:518]: Clean up resources, cleanup_instance: True, is_dangling: False
```
Contributor guide
Assessment
This issue has not been assessed yet.