dask / dask/dask-gateway

Using a Mapping config coupled to a KubeClusterConfig's environment and passing a non-string value cause a type error on startup

Open
#387 1 comment 0 reactions 0 assignees View on GitHub
codebase:dask-gateway-server enhancement
Dominant language
Python
Stars
148
Forks
93
PR merge metrics
No merged PRs in 30d

Description

Just noting a UX issue. If you use the mapping provided in https://github.com/dask/dask-gateway/pull/290, to set environment variables, you should ensure that the values are strings.

If not, kubernetes will throw an error (in the controller pod) and from the user's point of view things just hang indefinitely.

```
[I 2021-04-15 02:07:01.388 KubeController] Creating scheduler pod for cluster staging.671dcd6379c844f5af9375c25ebf22d7
[W 2021-04-15 02:07:01.391 KubeController] Error while reconciling cluster staging.671dcd6379c844f5af9375c25ebf22d7
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/controller.py", line 586, in reconciler_loop
requeue = await self.reconcile_cluster(name)
File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/controller.py", line 607, in reconcile_cluster
status_update, requeue = await self.handle_cluster(cluster)
File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/controller.py", line 643, in handle_cluster
return await self.handle_pending_cluster(cluster)
File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/controller.py", line 661, in handle_pending_cluster
sched_pod_name, sched_pod = await self.create_scheduler_pod_if_not_exists(
File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/controller.py", line 942, in create_scheduler_pod_if_not_exists
pod = await self.core_client.create_namespaced_pod(namespace, pod)
File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/utils.py", line 59, in func
return await method(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/api_client.py", line 180, in __call_api
response_data = await self.request(
File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/rest.py", line 229, in POST
return (await self.request("POST", url,
File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/rest.py", line 186, in request
raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.v1.EnvVar.Value: ReadString: expects \" or n, but found 1, error found in #10 byte of ...|\"value\": 1}, {\"name\"|..., bigger context ...|_DISTRIBUTED__WORKERS__RESOURCES__GPU\", \"value\": 1}, {\"name\": \"NVIDIA_DRIVER_CAPABILITIES\", \"value\":|...","reason":"BadRequest","code":400}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.