dask / dask/helm-chart

Network policies break daskhub

Open
#445 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
YAML
Stars
100
Forks
92
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:
It appears the default network policies from the jupyterhub helm chart breaks communication with dask-gateway and the kube-apiserver.

I deployed daskhub with default values onto a vanilla K8s cluster with a CNI that supports network policies (cilium).
`helm upgrade --install --create-namespace --namespace jhub01 jhub01 dask/daskhub`

With this deployment, the jupyterhub pod will not spawn and I receive this output:
![image](https://github.com/dask/helm-chart/assets/43788363/4a484716-b02f-4f58-a7d5-293f1922fcac)

Using hubble, I am able to see the packets are being dropped via network policy:
`hubble observe -n jhub01 -t drop -f`

> Feb 16 18:05:30.839: jhub01/hub-fc455bdb8-2n7ct:34144 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:31.862: jhub01/hub-fc455bdb8-2n7ct:34144 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:33.910: jhub01/hub-fc455bdb8-2n7ct:34144 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:35.043: jhub01/hub-fc455bdb8-2n7ct:38918 (ID:133898) <> XXX.XXX.XXX.148:6443 (kube-apiserver) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 18:05:36.086: jhub01/hub-fc455bdb8-2n7ct:38918 (ID:133898) <> XXX.XXX.XXX.148:6443 (kube-apiserver) Policy denied DROPPED (TCP Flags: SYN)

If I allow access to the kube-apiserver (reference ticket below), the pod will then spawn but I still get drops for dask-gateway communication:
>Feb 16 19:40:58.002: jhub01/jupyter-test:34604 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:00.498: jhub01/jupyter-test:53158 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:06.130: jhub01/jupyter-test:34604 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:07.936: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:08.950: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:10.998: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:14.914: jhub01/jupyter-test:43114 (ID:146419) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)
Feb 16 19:41:15.030: jhub01/hub-5fd4dbdb78-gmnvw:58384 (ID:133898) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-hzwrj:8000 (ID:170445) Policy denied DROPPED (TCP Flags: SYN)

Here is the list of network policies defined for the whole cluster:
`kubectl get networkpolicies.networking.k8s.io -A`

> NAMESPACE NAME POD-SELECTOR AGE
jhub01 hub app=jupyterhub,component=hub,release=jhub01 22h
jhub01 proxy app=jupyterhub,component=proxy,release=jhub01 22h
jhub01 singleuser app=jupyterhub,component=singleuser-server,release=jhub01 22h

Everything works when I deploy this network policy into the namespace:
````
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-ingress-egress
spec:
podSelector: {}
egress:
- {}
ingress:
- {}
policyTypes:
- Egress
- Ingress
````

**Anything else we need to know?**:
Bare Metal - K8s Server Version: v1.29.1
CRI-O Version: v1.29.1
Cilium Version: v1.15.1

This issue is related and is why we are seeing drops for the kube-apiserver:
- https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/3202

If I allow access to the kube-apiserver I then hit this issue:
- https://github.com/dask/helm-chart/issues/431

I think that the daskhub chart should deploy network policies to allow the jupyterhub pod to communicate with dask-gateway. Or perhaps something about the correct network policies should be documented since the default values don't allow dask-gateway communication?

**Environment**:

- Dask version: daskhub-2024.1.1
- Python version:
- Operating System: AlmaLinux 9
- Install method (conda, pip, source): helm

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.