jupyterhub / jupyterhub/systemdspawner

Use with LDAP authenticator

Open
#84 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
100
Forks
44
PR merge metrics
No merged PRs in 30d

Description

### Bug description
Setup queries LDAP server and launches a JupyterLab instance in the home directory of the user (each user has a Linux user set up already)

Without `systemdspawner` the setup works perfectly, but as soon as its configured spinning up the user's spawner hangs midway.

I suspect the user needs spawn home directory set in `systemdspawner`, and their permissions given to the conda environment's jlab executables?

(Corporate Usernames and IP addresses are masked in logs and jhubconfig.py)

### Set Up

- OS:
RHEL 8 on ppc64le architecture
- Version(s):
```
jupyter 1.0.0 py39hc1b9086_6 conda-forge
jupyter-lsp 1.4.1 pyhd8ed1ab_0 conda-forge
jupyter-resource-usage 0.6.0 pypi_0 pypi
jupyter-server-mathjax 0.2.3 pypi_0 pypi
jupyter-server-proxy 3.1.0 pypi_0 pypi
jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge
jupyter_console 6.4.0 pyhd8ed1ab_0 conda-forge
jupyter_core 4.7.1 py39hc1b9086_0 conda-forge
jupyter_server 1.10.2 pyhd8ed1ab_0 conda-forge
jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge
jupyterhub 1.4.2 py39hc1b9086_0 conda-forge
jupyterhub-base 1.4.2 py39hc1b9086_0 conda-forge
jupyterhub-idle-culler 1.1 pypi_0 pypi
jupyterhub-ldapauthenticator 1.3.2 py_0 conda-forge
jupyterhub-systemdspawner 0.15.0 pypi_0 pypi
jupyterlab 3.1.6 pyhd8ed1ab_0 conda-forge
jupyterlab-code-formatter 1.4.10 pypi_0 pypi
jupyterlab-drawio 0.9.0 pyhd8ed1ab_0 conda-forge
jupyterlab-git 0.32.0 pypi_0 pypi
jupyterlab-lsp 3.8.1 pyhd8ed1ab_0 conda-forge
jupyterlab-system-monitor 0.8.0 pypi_0 pypi
jupyterlab-topbar 0.6.1 pypi_0 pypi
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_server 2.7.0 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge
```

- Full environment

```
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
absl-py 0.13.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.7.4.post0 py39ha810350_0 conda-forge
alembic 1.6.5 pyhd8ed1ab_0 conda-forge
anyio 3.2.1 py39hc1b9086_0 conda-forge
appdirs 1.4.4 pypi_0 pypi
argon2-cffi 20.1.0 py39ha810350_2 conda-forge
astroid 2.6.2 pypi_0 pypi
async-timeout 3.0.1 py_1000 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 21.2.0 pyhd8ed1ab_0 conda-forge
autopep8 1.5.7 pyhd8ed1ab_0 conda-forge
babel 2.9.1 pyh44b312d_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bat 0.18.1 ha57a3a6_0 conda-forge
binutils_impl_linux-ppc64le 2.36.1 h5836da8_1 conda-forge
black 21.7b0 pypi_0 pypi
bleach 3.3.0 pyh44b312d_0 conda-forge
blinker 1.4 py_1 conda-forge
bokeh 2.3.3 pypi_0 pypi
brotlipy 0.7.0 py39ha810350_1001 conda-forge
bzip2 1.0.8 h4e0d66e_4 conda-forge
c-ares 1.17.1 h4e0d66e_1 conda-forge
ca-certificates 2021.5.30 h1084571_0 conda-forge
cachetools 4.2.2 pyhd8ed1ab_0 conda-forge
certifi 2021.5.30 py39hc1b9086_0 conda-forge
certipy 0.1.3 py_0 conda-forge
cffi 1.14.6 py39hfce1a3b_0 conda-forge
chardet 4.0.0 py39hc1b9086_1 conda-forge
click 8.0.1 py39hc1b9086_0 conda-forge
cloudpickle 1.6.0 pypi_0 pypi
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
configurable-http-proxy 4.4.0 node15_h42ff63e_0 conda-forge
cryptography 3.4.7 py39h1b86913_0 conda-forge
curl 7.78.0 h57b39c4_0 conda-forge
dataclasses 0.8 pyhc8e2a94_1 conda-forge
debugpy 1.3.0 py39had50986_0 conda-forge
decorator 5.0.9 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
distributed 2021.7.2 pypi_0 pypi
docker 5.0.0 pypi_0 pypi
dockerspawner 12.1.0 pypi_0 pypi
entrypoints 0.3 py39hde42818_1002 conda-forge
escapism 1.0.1 pypi_0 pypi
exa 0.10.1 ha57a3a6_0 conda-forge
expat 2.4.1 h3b9df90_0 conda-forge
flake8 3.9.2 pyhd8ed1ab_0 conda-forge
freetype 2.10.4 h8a70690_1 conda-forge
fsspec 2021.7.0 pypi_0 pypi
gcc_impl_linux-ppc64le 9.3.0 ha8fcf76_18 conda-forge
gettext 0.19.8.1 hf23c357_1005 conda-forge
git 2.32.0 pl5321h29e2a4d_0 conda-forge
gitdb 4.0.7 pypi_0 pypi
gitpython 3.1.18 pypi_0 pypi
google-auth 1.32.1 pyh6c4a22f_0 conda-forge
google-auth-oauthlib 0.4.1 py_2 conda-forge
greenlet 1.1.0 py39had50986_0 conda-forge
grpcio 1.38.1 py39hff9cdcc_0 conda-forge
heapdict 1.0.1 pypi_0 pypi
icu 68.1 hea85c5d_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 4.6.1 py39hc1b9086_0 conda-forge
ipykernel 6.0.1 py39h0de2273_0 conda-forge
ipython 7.25.0 py39h0de2273_1 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.6.3 pyhd3deb0d_0 conda-forge
isort 5.9.3 pyhd8ed1ab_0 conda-forge
jbig 2.1 h4e0d66e_2003 conda-forge
jedi 0.18.0 py39hc1b9086_2 conda-forge
jinja2 3.0.1 pyhd8ed1ab_0 conda-forge
jpeg 9d h339bb43_0 conda-forge
json5 0.9.5 pyh9f0ad1d_0 conda-forge
jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge
jupyter 1.0.0 py39hc1b9086_6 conda-forge
jupyter-lsp 1.4.1 pyhd8ed1ab_0 conda-forge
jupyter-resource-usage 0.6.0 pypi_0 pypi
jupyter-server-mathjax 0.2.3 pypi_0 pypi
jupyter-server-proxy 3.1.0 pypi_0 pypi
jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge
jupyter_console 6.4.0 pyhd8ed1ab_0 conda-forge
jupyter_core 4.7.1 py39hc1b9086_0 conda-forge
jupyter_server 1.10.2 pyhd8ed1ab_0 conda-forge
jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge
jupyterhub 1.4.2 py39hc1b9086_0 conda-forge
jupyterhub-base 1.4.2 py39hc1b9086_0 conda-forge
jupyterhub-idle-culler 1.1 pypi_0 pypi
jupyterhub-ldapauthenticator 1.3.2 py_0 conda-forge
jupyterhub-systemdspawner 0.15.0 pypi_0 pypi
jupyterlab 3.1.6 pyhd8ed1ab_0 conda-forge
jupyterlab-code-formatter 1.4.10 pypi_0 pypi
jupyterlab-drawio 0.9.0 pyhd8ed1ab_0 conda-forge
jupyterlab-git 0.32.0 pypi_0 pypi
jupyterlab-lsp 3.8.1 pyhd8ed1ab_0 conda-forge
jupyterlab-system-monitor 0.8.0 pypi_0 pypi
jupyterlab-topbar 0.6.1 pypi_0 pypi
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_server 2.7.0 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge
kernel-headers_linux-ppc64le 3.10.0 hbe0c576_10 conda-forge
krb5 1.19.1 haf43566_0 conda-forge
lazy-object-proxy 1.6.0 py39ha810350_0 conda-forge
lcms2 2.11 h6d9531b_1 conda-forge
ld_impl_linux-ppc64le 2.36.1 ha35d02b_1 conda-forge
ldap3 2.8.1 pyh9f0ad1d_0 conda-forge
lerc 2.2.1 h3b9df90_0 conda-forge
libblas 3.9.0 9_openblas conda-forge
libcblas 3.9.0 9_openblas conda-forge
libcurl 7.78.0 he415e40_0 conda-forge
libdeflate 1.7 h4e0d66e_5 conda-forge
libedit 3.1.20191231 h41a240f_2 conda-forge
libev 4.33 h6eb9509_1 conda-forge
libffi 3.3 hea85c5d_2 conda-forge
libgcc-devel_linux-ppc64le 9.3.0 h9f60fdd_18 conda-forge
libgcc-ng 9.3.0 hc7a5eb4_18 conda-forge
libgfortran-ng 9.3.0 hb3da1e1_18 conda-forge
libgfortran5 9.3.0 hb3da1e1_18 conda-forge
libgomp 9.3.0 hc7a5eb4_18 conda-forge
libiconv 1.16 h6eb9509_0 conda-forge
libidn2 2.3.1 h4e0d66e_0 conda-forge
liblapack 3.9.0 9_openblas conda-forge
libnghttp2 1.43.0 h42039ad_0 conda-forge
libopenblas 0.3.15 pthreads_h486567c_1 conda-forge
libpng 1.6.37 h38e1d09_2 conda-forge
libprotobuf 3.17.2 h690f14c_0 conda-forge
libsodium 1.0.18 h6eb9509_1 conda-forge
libssh2 1.9.0 ha5a9321_6 conda-forge
libstdcxx-ng 9.3.0 h22bc1b8_18 conda-forge
libtiff 4.3.0 h362c9b0_1 conda-forge
libunistring 0.9.10 h14c3975_0 conda-forge
libuuid 2.32.1 h4e0d66e_1000 conda-forge
libuv 1.41.1 h4e0d66e_0 conda-forge
libwebp-base 1.2.0 h4e0d66e_2 conda-forge
locket 0.2.1 pypi_0 pypi
lz4-c 1.9.3 h3b9df90_0 conda-forge
mako 1.1.4 pyh44b312d_0 conda-forge
markdown 3.3.4 pyhd8ed1ab_0 conda-forge
markupsafe 2.0.1 py39ha810350_0 conda-forge
matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge
mccabe 0.6.1 pypi_0 pypi
mistune 0.8.4 py39ha810350_1004 conda-forge
msgpack 1.0.2 pypi_0 pypi
multidict 5.1.0 py39ha810350_1 conda-forge
mypy-extensions 0.4.3 pypi_0 pypi
nbclassic 0.3.1 pyhd8ed1ab_1 conda-forge
nbclient 0.5.3 pyhd8ed1ab_0 conda-forge
nbconvert 6.1.0 py39hc1b9086_0 conda-forge
nbdime 3.1.0 pypi_0 pypi
nbformat 5.1.3 pyhd8ed1ab_0 conda-forge
ncurses 6.2 hea85c5d_4 conda-forge
nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge
nodejs 15.14.0 h4993e3a_0 conda-forge
notebook 6.4.0 pyha770c72_0 conda-forge
numpy 1.21.0 py39he089932_0 conda-forge
oauthlib 3.1.1 pyhd8ed1ab_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 h29f4549_1 conda-forge
openssl 1.1.1k h4e0d66e_0 conda-forge
packaging 21.0 pyhd8ed1ab_0 conda-forge
pamela 1.0.0 py_0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parso 0.8.2 pyhd8ed1ab_0 conda-forge
partd 1.2.0 pypi_0 pypi
pathspec 0.8.1 pypi_0 pypi
pcre2 10.37 h32277c3_0 conda-forge
perl 5.32.1 0_h4e0d66e_perl5 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py39hde42818_1002 conda-forge
pillow 8.3.1 py39hd2e0652_0 conda-forge
pip 21.2.4 pyhd8ed1ab_0 conda-forge
pluggy 0.13.1 py39hc1b9086_4 conda-forge
prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.19 pyha770c72_0 conda-forge
prompt_toolkit 3.0.19 hd8ed1ab_0 conda-forge
protobuf 3.17.2 py39had50986_0 conda-forge
psutil 5.8.0 pypi_0 pypi
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycodestyle 2.7.0 pyhd8ed1ab_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pycurl 7.43.0.6 py39h2a2158a_1 conda-forge
pydocstyle 6.1.1 pyhd8ed1ab_0 conda-forge
pyflakes 2.3.1 pyhd8ed1ab_0 conda-forge
pygments 2.9.0 pyhd8ed1ab_0 conda-forge
pyjwt 2.1.0 pyhd8ed1ab_0 conda-forge
pylint 2.9.3 pypi_0 pypi
pynvml 11.0.0 pypi_0 pypi
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyrsistent 0.17.3 py39ha810350_2 conda-forge
pysocks 1.7.1 py39hc1b9086_3 conda-forge
python 3.9.6 h82ac395_0_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-editor 1.0.4 py_0 conda-forge
python-json-logger 2.0.1 pyh9f0ad1d_0 conda-forge
python-lsp-jsonrpc 1.0.0 pyhd8ed1ab_0 conda-forge
python-lsp-server 1.2.1 pypi_0 pypi
python_abi 3.9 2_cp39 conda-forge
pytz 2021.1 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 5.4.1 pypi_0 pypi
pyzmq 22.1.0 py39h050fd71_0 conda-forge
readline 8.1 h5c45dff_0 conda-forge
regex 2021.7.6 pypi_0 pypi
requests 2.25.1 pyhd3deb0d_0 conda-forge
requests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forge
requests-unixsocket 0.2.0 py_0 conda-forge
retrolab 0.2.2 pypi_0 pypi
rope 0.19.0 pyhd8ed1ab_0 conda-forge
rsa 4.7.2 pyh44b312d_0 conda-forge
ruamel.yaml 0.17.10 py39ha810350_0 conda-forge
ruamel.yaml.clib 0.2.2 py39ha810350_2 conda-forge
rust 1.53.0 h978bb50_0 conda-forge
rust-std-powerpc64le-unknown-linux-gnu 1.53.0 h6ac4ec8_0 conda-forge
send2trash 1.7.1 pyhd8ed1ab_0 conda-forge
setuptools 49.6.0 py39hc1b9086_3 conda-forge
simpervisor 0.4 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
smmap 4.0.0 pypi_0 pypi
sniffio 1.2.0 py39hc1b9086_1 conda-forge
snowballstemmer 2.1.0 pyhd8ed1ab_0 conda-forge
sortedcontainers 2.4.0 pypi_0 pypi
sqlalchemy 1.4.20 py39ha810350_0 conda-forge
sqlite 3.36.0 h4e2196e_0 conda-forge
sysroot_linux-ppc64le 2.17 h8b29623_10 conda-forge
tblib 1.7.0 pypi_0 pypi
tensorboard 2.4.1 pyhd8ed1ab_0 conda-forge
tensorboard-plugin-wit 1.8.0 pyh44b312d_0 conda-forge
terminado 0.10.1 py39hc1b9086_0 conda-forge
testpath 0.5.0 pyhd8ed1ab_0 conda-forge
tk 8.6.10 h38e1d09_1 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 1.2.1 pypi_0 pypi
toolz 0.11.1 pypi_0 pypi
tornado 6.1 py39ha810350_1 conda-forge
traitlets 5.0.5 py_0 conda-forge
typing-extensions 3.10.0.0 hd8ed1ab_0 conda-forge
typing_extensions 3.10.0.0 pyha770c72_0 conda-forge
tzdata 2021a he74cb21_1 conda-forge
ujson 4.0.2 py39had50986_0 conda-forge
urllib3 1.26.6 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 0.57.0 py39hc1b9086_4 conda-forge
werkzeug 2.0.1 pyhd8ed1ab_0 conda-forge
wget 1.20.3 ha5a9321_0 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
widgetsnbextension 3.5.1 py39hc1b9086_4 conda-forge
wrapt 1.12.1 py39ha810350_3 conda-forge
xeus 1.0.3 h1916c12_1 conda-forge
xeus-python 0.12.5 py39h1f10ff4_2 conda-forge
xz 5.2.5 h6eb9509_1 conda-forge
yapf 0.31.0 pyhd8ed1ab_0 conda-forge
yarl 1.6.3 py39ha810350_2 conda-forge
zeromq 4.3.4 h3b9df90_0 conda-forge
zict 2.0.0 pypi_0 pypi
zipp 3.5.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h6eb9509_1010 conda-forge
zstd 1.5.0 h65c4b1a_0 conda-forge
```

- Configuration

```python
import sys
c.Spawner.http_timeout = 120
c.Spawner.start_timeout = 120
c.Spawner.cmd=["jupyter-labhub"]
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address = **********
c.LDAPAuthenticator.bind_dn_template = [
"*******\{username}",
]
c.LDAPAuthenticator.use_ssl = True
c.LDAPAuthenticator.server_port = 636
c.LDAPAuthenticator.valid_username_regex = ".*"
c.JupyterHub.services = [
{
'name': 'idle-culler',
'admin': True,
'command': [
sys.executable,
'-m', 'jupyterhub_idle_culler',
'--timeout=72000'
],
}
]
c.JupyterHub.admin_access = True
c.JupyterHub.bind_url = 'http://:8888'
c.JupyterHub.hub_connect_ip = '0.0.0.0'
c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
```

- Logs

```console

[I 2021-08-14 09:52:37.131 JupyterHub log:189] 302 GET /user/*******/lab -> /hub/user/*******/lab (@********) 0.69ms
[W 2021-08-14 09:52:37.348 JupyterHub base:390] Invalid or expired cookie token
[I 2021-08-14 09:52:37.349 JupyterHub log:189] 302 GET /hub/user/********/lab -> /hub/login?next=%2Fhub%2Fuser%2F********%2Flab (@********) 0.99ms
[I 2021-08-14 09:52:37.568 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2Fuser%********%2Flab (@********) 42.46ms
[D 2021-08-14 09:52:42.034 JupyterHub ldapauthenticator:379] Attempting to bind ******** with ********\********
[D 2021-08-14 09:52:42.383 JupyterHub ldapauthenticator:392] Status of user bind ******** with ********\******** : True
[D 2021-08-14 09:52:42.388 JupyterHub base:530] Setting cookie for ********: jupyterhub-hub-login
[D 2021-08-14 09:52:42.388 JupyterHub base:526] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2021-08-14 09:52:42.388 JupyterHub base:762] User logged in: ********
[I 2021-08-14 09:52:42.389 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2Fuser%********%2Flab -> /hub/user/********/lab (********@********) 355.25ms
[D 2021-08-14 09:52:42.678 JupyterHub user:317] Creating for ********:
[D 2021-08-14 09:52:42.679 JupyterHub systemdspawner:155] user:******** Initialized spawner with unit jupyter-********-singleuser
[E 2021-08-14 09:52:42.690 JupyterHub log:189] 503 GET /hub/user/********/lab (********@********) 22.47ms
[D 2021-08-14 09:52:42.984 JupyterHub log:189] 200 GET /hub/static/js/not_running.js?v=20210814095231 (@********) 7.36ms
[D 2021-08-14 09:52:43.172 JupyterHub log:189] 200 GET /hub/static/js/utils.js?v=20210814095231 (@********) 2.46ms
[D 2021-08-14 09:52:45.604 JupyterHub pages:217] Triggering spawn with default options for ********
[D 2021-08-14 09:52:45.604 JupyterHub base:880] Initiating spawn for ********
[D 2021-08-14 09:52:45.604 JupyterHub base:884] 0/100 concurrent spawns
[D 2021-08-14 09:52:45.604 JupyterHub base:889] 0 active servers
[I 2021-08-14 09:52:45.619 JupyterHub provider:574] Creating oauth client jupyterhub-user-********
[D 2021-08-14 09:52:45.640 JupyterHub user:631] Calling Spawner.start for ********
[D 2021-08-14 09:52:45.640 JupyterHub systemdspawner:201] user:******** Using port 45213 to start spawning user server
Running as unit: jupyter-********-singleuser.service
[I 2021-08-14 09:52:46.607 JupyterHub log:189] 302 GET /hub/spawn/********?next=%2Fhub%2Fuser%2F********%2Flab -> /hub/spawn-pending/********?next=%2Fhub%2Fuser%2F********%2Flab (********@********) 1005.57ms
[I 2021-08-14 09:52:46.864 JupyterHub pages:402] ******** is pending spawn
[I 2021-08-14 09:52:46.869 JupyterHub log:189] 200 GET /hub/spawn-pending/********?next=%2Fhub%2Fuser%********%2Flab (********@********) 7.68ms
[W 2021-08-14 09:52:55.606 JupyterHub base:1008] User ******** is slow to start (timeout=10)

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.