jupyterhub / jupyterhub/batchspawner
internal_ssl + SlurmSpawner leads to certificate verification error
- Dominant language
- Python
- Stars
- 207
- Forks
- 136
- PR merge metrics
- No merged PRs in 30d
Description
### Bug description
I have setup a JupyterHub instance on my cluster's login node that uses `SlurmSpawner` to spawn notebook servers on our cluster. I have verified that `SlurmSpawner` works (wonderfully btw) and that SSL works everywhere except between the Hub server and the spawned notebook servers. I was experimenting with JupyterHub's `internal_ssl` feature but as soon as I set it to `True` in the config I was met with this error
```
[W 2020-09-19 20:15:21.818 SingleUserNotebookApp iostream:1432] SSL Error on 9 ('[IP]', 8081): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)
[E 2020-09-19 20:15:21.819 SingleUserNotebookApp singleuser:434] Failed to connect to my Hub at https://[IP]:8081/hub/api (attempt 3/5). Is it running?
Traceback (most recent call last):
File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/singleuser.py", line 432, in check_hub_version
resp = await client.fetch(self.hub_api_url)
File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/simple_httpclient.py", line 330, in run
stream = await self.tcp_client.connect(
File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/tcpclient.py", line 293, in connect
stream = await stream.start_tls(
File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/iostream.py", line 1417, in _do_ssl_handshake
self.socket.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108
```
I have looked at #31, #103, and jupyterhub/jupyterhub#2055 but I cannot find good documentation on this issue/what I am doing wrong.
### Your personal set up
JupyterHub instance using `SlurmSpawner` to spawn notebook servers. Hub instance is on the same machine as Slurm login node.
- OS:
Ubuntu 20.04.1 LTS (all nodes)
- Version:
```
jupyter core : 4.6.3
jupyter-notebook : 6.1.4
qtconsole : not installed
ipython : 7.18.1
ipykernel : 5.3.4
jupyter client : 6.1.7
jupyter lab : 2.2.8
nbconvert : 6.0.3
ipywidgets : 7.5.1
nbformat : 5.0.7
traitlets : 5.0.4
```
- Configuration:
`jupyterhub.XXX.XXX` is CNAME-d to a www server on our network and all traffic is proxied through the www server to the login node (where jupyterhub is hosted). `infocube.XXX.XXX.XXX` is the login node.
`jupyterhub_config.py`:
https://pastebin.com/BJRb3NfP
Contributor guide
Assessment
This issue has not been assessed yet.