jupyterhub / jupyterhub/batchspawner

Jupytehrhub not able to talk to notebook running in the compute node (SLURM)

Open
#233 16 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
207
Forks
136
PR merge metrics
No merged PRs in 30d

Description

### Bug description
When I strart a server the jobs gets scheduled and successfully starts but the hub cannot talk to it.

#### Expected behaviour
The hub knows about the server running in the node and the server starts.

#### Actual behaviour
From the job logs:
```
[I 2022-03-31 15:53:10.848 SingleUserLabApp serverapp:2674] Serving notebooks from local directory: /home/bria
[I 2022-03-31 15:53:10.848 SingleUserLabApp serverapp:2674] Jupyter Server 1.15.6 is running at:
[I 2022-03-31 15:53:10.848 SingleUserLabApp serverapp:2674] http://computenode.domain.com:59640/user/arnau.bria@domain.com/lab
[I 2022-03-31 15:53:10.849 SingleUserLabApp serverapp:2674] or http://127.0.0.1:59640/user/arnau.bria@domain.com/lab
[I 2022-03-31 15:53:10.849 SingleUserLabApp serverapp:2675] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2022-03-31 15:53:10.857 SingleUserLabApp mixins:596] Updating Hub with activity every 300 seconds
[D 2022-03-31 15:53:10.857 SingleUserLabApp mixins:558] Notifying Hub of activity 2022-03-31T13:53:10.600560Z
```
From the hub logs:
```
Mar 31 15:53:02 hubserver jupyterhub.service: [D 2022-03-31 15:53:02.528 JupyterHub batchspawner:288] Spawner querying job: sudo -E -u arnau.bria@domain.com squeue -h -j 14270408 -o '%T %B'
Mar 31 15:53:02 hubserver systemd: Created slice User Slice of bria.
Mar 31 15:53:02 hubserver systemd: Starting User Slice of bria.
Mar 31 15:53:02 hubserver systemd: Started Session c2756 of user bria.
Mar 31 15:53:02 hubserver systemd: Starting Session c2756 of user bria.
Mar 31 15:53:02 hubserver systemd: Removed slice User Slice of bria.
Mar 31 15:53:02 hubserver systemd: Stopping User Slice of bria.
Mar 31 15:53:06 hubserver jupyterhub.service: [D 2022-03-31 15:53:06.512 JupyterHub base:281] Recording first activity for
Mar 31 15:53:06 hubserver jupyterhub.service: [D 2022-03-31 15:53:06.517 JupyterHub scopes:301] Authenticated with token
Mar 31 15:53:06 hubserver jupyterhub.service: [I 2022-03-31 15:53:06.520 JupyterHub log:189] 200 POST /hub/api/batchspawner (arnau.bria@domain.com@148.191.97.103) 10.94ms
Mar 31 15:53:10 hubserver jupyterhub.service: [I 2022-03-31 15:53:10.848 JupyterHub log:189] 200 GET /hub/api (@148.191.97.103) 0.85ms
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.861 JupyterHub scopes:301] Authenticated with token
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.862 JupyterHub scopes:491] Checking access via scope users:activity
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.862 JupyterHub scopes:402] Argument-based access to /hub/api/users/arnau.bria@domain.com/activity via users:activity
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.864 JupyterHub users:855] Activity for user arnau.bria@domain.com: 2022-03-31T13:53:10.600560Z
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.864 JupyterHub users:873] Activity on server arnau.bria@domain.com/: 2022-03-31T13:53:10.600560Z
Mar 31 15:53:10 hubserver jupyterhub.service: [I 2022-03-31 15:53:10.870 JupyterHub log:189] 200 POST /hub/api/users/arnau.bria@domain.com/activity (arnau.bria@domain.com@148.191.97.103) 11.01ms
```

### How to reproduce
Request a job using the batchspawner.

### Your personal set up

- No firewaal between hubserver - computenode.
- Software installed with EasyBuild
- OS: Scientific Linux 7.9
- Version(s):
```

alembic 1.7.7
anyio 0.0.0
argon2-cffi 21.3.0
argon2-cffi-bindings 0.0.0
asttokens 2.0.5
async-generator 1.10
attrs 21.4.0
Babel 2.9.1
backcall 0.2.0
batchspawner 1.1.0
beautifulsoup4 4.10.0
bleach 4.1.0
certifi 2021.10.8
certipy 0.1.3
cffi 1.15.0
charset-normalizer 2.0.12
colorama 0.4.4
cryptography 36.0.2
debugpy 1.5.1
decorator 5.1.1
defusedxml 0.7.1
deprecation 2.1.0
entrypoints 0.4
executing 0.0.0
flit_core 3.6.0
gitdb 4.0.9
GitPython 3.1.27
greenlet 1.1.2
idna 3.3
iniconfig 0.0.0
ipykernel 6.9.2
ipython 8.1.1
ipython-genutils 0.2.0
ipywidgets 7.6.5
jedi 0.18.1
Jinja2 3.0.3
json5 0.9.6
jsonschema 0.0.0
jupyter-client 7.1.2
jupyter-core 4.9.2
jupyter-packaging 0.11.1
jupyter-resource-usage 0.6.1
jupyter-server 1.15.6
jupyter-server-mathjax 0.2.3
jupyter-telemetry 0.1.0
jupyterhub 2.2.2
jupyterlab 3.3.2
jupyterlab-git 0.34.2
jupyterlab-launcher 0.13.1
jupyterlab-pygments 0.1.2
jupyterlab-server 2.11.2
jupyterlab-widgets 1.0.2
Mako 1.2.0
MarkupSafe 2.1.1
matplotlib-inline 0.1.3
mistune 0.8.4
nbclassic 0.3.7
nbclient 0.5.13
nbconvert 6.4.4
nbdime 3.1.1
nbformat 5.2.0
nest-asyncio 1.5.4
notebook 6.4.10
notebook-shim 0.1.0
oauthenticator 14.2.0
oauthlib 3.2.0
packaging 21.3
pamela 1.0.0
pandocfilters 1.5.0
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pip 22.0.4
pluggy 0.0.0
poetry 1.1.13
poetry-core 1.0.8
prometheus-client 0.13.1
prompt-toolkit 3.0.28
psutil 5.9.0
ptyprocess 0.7.0
pure-eval 0.0.0
py 1.10.0
pycparser 2.21
Pygments 2.11.2
PyJWT 2.3.0
pyOpenSSL 22.0.0
pyparsing 3.0.7
pyrsistent 0.18.1
pytest 0.0.0
python-dateutil 0.0.0
python-json-logger 2.0.2
pytz 2022.1
pyzmq 22.3.0
requests 2.27.1
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.6
semantic-version 2.9.0
Send2Trash 1.8.0
setuptools 60.10.0
setuptools-rust 1.1.2
six 1.16.0
smmap 3.0.1
sniffio 1.2.0
soupsieve 2.3.1
SQLAlchemy 1.4.32
stack-data 0.0.0
terminado 0.13.3
testpath 0.6.0
toml 0.10.2
tomlkit 0.10.0
tornado 6.1
traitlets 5.1.1
typing-extensions 3.10.0.2
urllib3 1.26.9
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.3.1
wheel 0.37.1
wrapspawner 1.0.1
```

- Configuration

```python
# Configuration file for jupyterhub.
c.Spawner.env_keep = [ 'PATH',
'PYTHONPATH',
'CONDA_ROOT',
'CONDA_DEFAULT_ENV',
'VIRTUAL_ENV',
'LANG',
'LC_ALL',
'R_LIBS_SITE',
'LD_LIBRARY_PATH',
'LIBRARY_PATH',
'SSL_CERT_DIR',
'CA_BUNDLE',
'SSL_CERT_FILE',
'JUPYTERLAB_DIR',
'JUPYTER_CONFIG_PATH',
'JUPYTER_CONFIG_DIR',
'JUPYTERHUB_API_URL',
'JUPYTERHUB_BASE_URL',
'JUPYTERHUB_CLIENT_ID',
'JUPYTERHUB_OAUTH_CALLBACK_URL',
'JUPYTERHUB_SERVER_NAME',
'JUPYTERHUB_SERVICE_PREFIX',
'JUPYTERHUB_USER'
]

# Uncomment if you want to debug
c.Spawner.args = ['--debug' ]
c.Application.log_level = 'DEBUG'
c.ConfigurableHTTPProxy.debug = True

# Configurable proxy
c.JupyterHub.proxy_class = 'jupyterhub.proxy.ConfigurableHTTPProxy'
c.JupyterHub.ssl_cert = '/etc/ssl/hubserver.domain.com.cert'
c.JupyterHub.ssl_key = '/etc/ssl/hubserver.domain.com.key'

# OAuth
c.PAMAuthenticator.open_sessions = False
import os
from oauthenticator.azuread import AzureAdOAuthenticator
c.JupyterHub.authenticator_class = AzureAdOAuthenticator

c.AzureAdOAuthenticator.tenant_id = os.environ.get('AAD_TENANT_ID')

c.AzureAdOAuthenticator.oauth_callback_url = 'https://inh-bio-jupyterhub-test.domain.com/hub/oauth_callback'
c.AzureAdOAuthenticator.client_id = '123'
c.AzureAdOAuthenticator.client_secret = 'secret'
c.AzureAdOAuthenticator.scope = ['openid']
c.AzureAdOAuthenticator.username_claim = 'preferred_username'

# Multiple servers
c.JupyterHub.allow_named_servers = True
c.JupyterHub.named_server_limit_per_user = 5

##BatchSpawner
#
# SLURM~# this is needed otherwise it does not use fqdn
c.SlurmSpawner.state_exechost_exp = r'\1.domain.com'
import batchspawner
c.Spawner.start_timeout=300
c.JupyterHub.bind_url = 'https://inh-bio-jupyterhub-test.domain.com/'
#c.JupyterHub.default_url = 'home'
c.JupyterHub.hub_ip = 'inh-bio-jupyterhub-test.domain.com'
#
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 300
c.ProfilesSpawner.profiles = [
("Local server", 'local', 'jupyterhub.spawner.LocalProcessSpawner', {'ip':'0.0.0.0'} ),
('Interactive Cluster - 2 cores, 4 GB, 8 hours', 'lab', 'batchspawner.SlurmSpawner',
dict(req_nprocs='2', req_partition='cpu',req_prologue='ml load Python/3.9.5-GCCcore-10.3.0-jupyter',req_keepvars='ALL')),
('Interactive Cluster - 12 cores, 48 GB, 8 hours', 'lab', 'batchspawner.SlurmSpawner',
dict(req_nprocs='12', req_partition='cpu',req_prologue='ml load Python/3.9.5-GCCcore-10.3.0-jupyter',req_keepvars='ALL')),
]

```

- Logs

Hub logs (148.191.97.103 resolves to computenode).
```
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.103 JupyterHub scopes:491] Checking access via scope servers
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.103 JupyterHub scopes:402] Argument-based access to /hub/spawn via servers
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.104 JupyterHub pages:255] Triggering spawn with supplied form options for arna
u.bria@domain.com
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.104 JupyterHub base:931] Initiating spawn for arnau.bria@domain.
com
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.104 JupyterHub base:935] 0/100 concurrent spawns
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.104 JupyterHub base:940] 0 active servers
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.107 JupyterHub roles:477] Checking token permissions against requested role se
rver
Mar 31 15:52:49 hubserver jupyterhub.service: [I 2022-03-31 15:52:49.108 JupyterHub roles:482] Adding role server to token:
Mar 31 15:52:49 hubserver jupyterhub.service: [I 2022-03-31 15:52:49.120 JupyterHub provider:607] Creating oauth client jupyterhub-user-arnau.bria%
40domain.com
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.136 JupyterHub user:728] Calling Spawner.start for arnau.bria@domain.com
Mar 31 15:52:49 hubserver jupyterhub.service: [W 2022-03-31 15:52:49.138 JupyterHub spawner:230] Setting Spawner.server for {self._log_name} with n
o underlying orm_spawner
Mar 31 15:52:49 hubserver jupyterhub.service: [I 2022-03-31 15:52:49.149 JupyterHub batchspawner:262] Spawner submitting job using sudo -E -u arnau
.bria@domain.com sbatch --parsable
Mar 31 15:52:49 hubserver jupyterhub.service: [I 2022-03-31 15:52:49.149 JupyterHub batchspawner:263] Spawner submitted script:
Mar 31 15:52:49 hubserver jupyterhub.service: #!/bin/bash
Mar 31 15:52:49 hubserver jupyterhub.service: #SBATCH --output=/home/bria/jupyterhub_slurmspawner_%j.log
Mar 31 15:52:49 hubserver jupyterhub.service: #SBATCH --job-name=spawner-jupyterhub
Mar 31 15:52:49 hubserver jupyterhub.service: #SBATCH --chdir=/home/bria
Mar 31 15:52:49 hubserver jupyterhub.service: #SBATCH --export=ALL
Mar 31 15:52:49 hubserver jupyterhub.service: #SBATCH --get-user-env=L
Mar 31 15:52:49 hubserver jupyterhub.service: #SBATCH --partition=cpu
Mar 31 15:52:49 hubserver jupyterhub.service: #SBATCH --cpus-per-task=2
Mar 31 15:52:49 hubserver jupyterhub.service: set -euo pipefail
Mar 31 15:52:49 hubserver jupyterhub.service: trap 'echo SIGTERM received' TERM
Mar 31 15:52:49 hubserver jupyterhub.service: ml load Python/3.9.5-GCCcore-10.3.0-jupyter
Mar 31 15:52:49 hubserver jupyterhub.service: which jupyterhub-singleuser
Mar 31 15:52:49 hubserver jupyterhub.service: srun batchspawner-singleuser jupyterhub-singleuser --debug
Mar 31 15:52:49 hubserver jupyterhub.service: echo "jupyterhub-singleuser ended gracefully"
Mar 31 15:52:49 hubserver systemd: Created slice User Slice of bria.
Mar 31 15:52:49 hubserver systemd: Starting User Slice of bria.
Mar 31 15:52:49 hubserver systemd: Started Session c2731 of user bria.
Mar 31 15:52:49 hubserver systemd: Starting Session c2731 of user bria.
Mar 31 15:52:49 hubserver jupyterhub.service: [I 2022-03-31 15:52:49.240 JupyterHub batchspawner:266] Job submitted. cmd: sudo -E -u arnau.bria@domain.com sbatch --parsable output: 14270408
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.242 JupyterHub batchspawner:288] Spawner querying job: sudo -E -u arnau.bria@domain.com squeue -h -j 14270408 -o '%T %B'
Mar 31 15:52:49 hubserver systemd: Removed slice User Slice of bria.
Mar 31 15:52:49 hubserver systemd: Stopping User Slice of bria.
Mar 31 15:52:49 hubserver systemd: Created slice User Slice of bria.
Mar 31 15:52:49 hubserver systemd: Starting User Slice of bria.
Mar 31 15:52:49 hubserver systemd: Started Session c2732 of user bria.
Mar 31 15:52:49 hubserver systemd: Starting Session c2732 of user bria.
Mar 31 15:52:49 hubserver jupyterhub.service: [D 2022-03-31 15:52:49.291 JupyterHub batchspawner:394] Job 14270408 still pending
[... once the job starts...]
Mar 31 15:53:02 hubserver systemd: Created slice User Slice of bria.
Mar 31 15:53:02 hubserver systemd: Starting User Slice of bria.
Mar 31 15:53:02 hubserver systemd: Started Session c2756 of user bria.
Mar 31 15:53:02 hubserver systemd: Starting Session c2756 of user bria.
Mar 31 15:53:02 hubserver systemd: Removed slice User Slice of bria.
Mar 31 15:53:02 hubserver systemd: Stopping User Slice of bria.
Mar 31 15:53:06 hubserver jupyterhub.service: [D 2022-03-31 15:53:06.512 JupyterHub base:281] Recording first activity for
Mar 31 15:53:06 hubserver jupyterhub.service: [D 2022-03-31 15:53:06.517 JupyterHub scopes:301] Authenticated with token
Mar 31 15:53:06 hubserver jupyterhub.service: [I 2022-03-31 15:53:06.520 JupyterHub log:189] 200 POST /hub/api/batchspawner (arnau.bria@domain.com@148.191.97.103) 10.94ms
Mar 31 15:53:10 hubserver jupyterhub.service: [I 2022-03-31 15:53:10.848 JupyterHub log:189] 200 GET /hub/api (@148.191.97.103) 0.85ms
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.861 JupyterHub scopes:301] Authenticated with token
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.862 JupyterHub scopes:491] Checking access via scope users:activity
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.862 JupyterHub scopes:402] Argument-based access to /hub/api/users/arnau.bria@
domain.com/activity via users:activity
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.864 JupyterHub users:855] Activity for user arnau.bria@domain.co
m: 2022-03-31T13:53:10.600560Z
Mar 31 15:53:10 hubserver jupyterhub.service: [D 2022-03-31 15:53:10.864 JupyterHub users:873] Activity on server arnau.bria@domain.c
om/: 2022-03-31T13:53:10.600560Z
Mar 31 15:53:10 hubserver jupyterhub.service: [I 2022-03-31 15:53:10.870 JupyterHub log:189] 200 POST /hub/api/users/arnau.bria@domain.com/activity (arnau.bria@domain.com@148.191.97.103) 11.01ms
Mar 31 15:57:19 hubserver jupyterhub.service: [D 2022-03-31 15:57:19.071 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
Mar 31 15:57:19 hubserver jupyterhub.service: 15:57:19.073 [ConfigProxy] #033[32minfo#033[39m: 200 GET /api/routes
Mar 31 15:57:19 hubserver jupyterhub.service: [D 2022-03-31 15:57:19.074 JupyterHub proxy:346] Checking routes
Mar 31 15:57:49 hubserver jupyterhub.service: [W 2022-03-31 15:57:49.144 JupyterHub user:807] arnau.bria@domain.com's server failed to start in 300 seconds, giving up.
Mar 31 15:57:49 hubserver jupyterhub.service: Common causes of this timeout, and debugging tips:
Mar 31 15:57:49 hubserver jupyterhub.service: 1. Everything is working, but it took too long.
Mar 31 15:57:49 hubserver jupyterhub.service: To fix: increase `Spawner.start_timeout` configuration
Mar 31 15:57:49 hubserver jupyterhub.service: to a number of seconds that is enough for spawners to finish starting.
Mar 31 15:57:49 hubserver jupyterhub.service: 2. The server didn't finish starting,
Mar 31 15:57:49 hubserver jupyterhub.service: or it crashed due to a configuration issue.
Mar 31 15:57:49 hubserver jupyterhub.service: Check the single-user server's logs for hints at what needs fixing.
Mar 31 15:57:49 hubserver jupyterhub.service: [D 2022-03-31 15:57:49.144 JupyterHub user:913] Stopping arnau.bria@domain.com
Mar 31 15:57:49 hubserver jupyterhub.service: [D 2022-03-31 15:57:49.145 JupyterHub batchspawner:288] Spawner querying job: sudo -E -u arnau.bria@domain.com squeue -h -j 14270408 -o '%T %B'
Mar 31 15:57:49 hubserver systemd: Created slice User Slice of bria.
Mar 31 15:57:49 hubserver systemd: Starting User Slice of bria.
Mar 31 15:57:49 hubserver systemd: Started Session c2757 of user bria.
Mar 31 15:57:49 hubserver systemd: Starting Session c2757 of user bria.
Mar 31 15:57:49 hubserver systemd: Removed slice User Slice of bria.
Mar 31 15:57:49 hubserver jupyterhub.service: [I 2022-03-31 15:57:49.223 JupyterHub batchspawner:431] Stopping server job 14270408
```

Job logs:

```
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/bin/jupyterhub-singleuser
[I 2022-03-31 15:53:09.658 SingleUserLabApp mixins:614] Starting jupyterhub single-user server version 2.2.2
[I 2022-03-31 15:53:09.658 SingleUserLabApp mixins:628] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.3.2
[I 2022-03-31 15:53:09.658 SingleUserLabApp mixins:628] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.15.6
[D 2022-03-31 15:53:09.673 SingleUserLabApp application:174] Searching ['/home/bria', '/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter', '/home/bria/.jupyter', '/home/bria/.local/etc/jupyter', '/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2022-03-31 15:53:09.673 SingleUserLabApp application:731] Looking for jupyter_config in /etc/jupyter
[D 2022-03-31 15:53:09.673 SingleUserLabApp application:731] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2022-03-31 15:53:09.673 SingleUserLabApp application:731] Looking for jupyter_config in /apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter
[D 2022-03-31 15:53:09.674 SingleUserLabApp application:731] Looking for jupyter_config in /home/bria/.local/etc/jupyter
[D 2022-03-31 15:53:09.674 SingleUserLabApp application:731] Looking for jupyter_config in /home/bria/.jupyter
[D 2022-03-31 15:53:09.674 SingleUserLabApp application:731] Looking for jupyter_config in /apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter
[D 2022-03-31 15:53:09.674 SingleUserLabApp application:731] Looking for jupyter_config in /home/bria
[D 2022-03-31 15:53:09.677 SingleUserLabApp application:731] Looking for jupyter_server_config in /etc/jupyter
[D 2022-03-31 15:53:09.677 SingleUserLabApp application:731] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2022-03-31 15:53:09.677 SingleUserLabApp application:731] Looking for jupyter_server_config in /apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter
[D 2022-03-31 15:53:09.678 SingleUserLabApp application:731] Looking for jupyter_server_config in /home/bria/.local/etc/jupyter
[D 2022-03-31 15:53:09.679 SingleUserLabApp application:731] Looking for jupyter_server_config in /home/bria/.jupyter
[D 2022-03-31 15:53:09.679 SingleUserLabApp application:731] Looking for jupyter_server_config in /apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter
[D 2022-03-31 15:53:09.679 SingleUserLabApp application:731] Looking for jupyter_server_config in /home/bria
[D 2022-03-31 15:53:09.683 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_server_config:
/etc/jupyter/jupyter_server_config.json
[D 2022-03-31 15:53:09.683 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_server_config:
/usr/local/etc/jupyter/jupyter_server_config.json
[D 2022-03-31 15:53:09.684 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_server_config:
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyter_resource_usage.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyter_server_mathjax.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyterlab.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyterlab_git.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/nbclassic.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/nbdime.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/notebook_shim.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.json
[D 2022-03-31 15:53:09.691 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_server_config:
/home/bria/.local/etc/jupyter/jupyter_server_config.json
[D 2022-03-31 15:53:09.691 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_server_config:
/home/bria/.jupyter/jupyter_server_config.json
[D 2022-03-31 15:53:09.692 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_server_config:
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyter_resource_usage.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyter_server_mathjax.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyterlab.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyter_server_mathjax.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyterlab.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/jupyterlab_git.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/nbclassic.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/nbdime.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.d/notebook_shim.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_server_config.json
[D 2022-03-31 15:53:09.694 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_server_config:
/home/bria/jupyter_server_config.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/lib/python3.9/site-packages/jupyter_server_mathjax/app.py:40: FutureWarning: The alias `_()` will be deprecated. Use `_i18n()` instead.
help=_("""The MathJax.js configuration file that is to be used."""),
[I 2022-03-31 15:53:09.825 SingleUserLabApp manager:345] jupyter_resource_usage | extension was successfully linked.
[D 2022-03-31 15:53:09.832 MathJaxExtension] Config changed: {'FileContentsManager': {'delete_to_trash': False}, 'SingleUserLabApp': {'port': 59640, 'log_level': 'DEBUG'}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'ServerApp': {'jpserver_extensions': }}
[D 2022-03-31 15:53:09.833 SingleUserLabApp application:326] Config changed: {'FileContentsManager': {'delete_to_trash': False}, 'SingleUserLabApp': {'port': 59640, 'log_level': 'DEBUG'}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'ServerApp': {'jpserver_extensions': }}
[I 2022-03-31 15:53:09.834 SingleUserLabApp manager:345] jupyter_server_mathjax | extension was successfully linked.
[D 2022-03-31 15:53:09.847 LabApp] Config changed: {'NotebookApp': {}, 'ServerApp': {'jpserver_extensions': }, 'FileContentsManager': {'delete_to_trash': False}, 'SingleUserLabApp': {'port': 59640, 'log_level': 'DEBUG'}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[I 2022-03-31 15:53:09.848 SingleUserLabApp manager:345] jupyterlab | extension was successfully linked.
[I 2022-03-31 15:53:09.848 SingleUserLabApp manager:345] jupyterlab_git | extension was successfully linked.
[D 2022-03-31 15:53:09.858 NotebookApp] Config changed: {'NotebookApp': {}, 'ServerApp': {'jpserver_extensions': }, 'FileContentsManager': {'delete_to_trash': False}, 'SingleUserLabApp': {'port': 59640, 'log_level': 'DEBUG'}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[I 2022-03-31 15:53:09.859 SingleUserLabApp manager:345] nbclassic | extension was successfully linked.
[I 2022-03-31 15:53:09.859 SingleUserLabApp manager:345] nbdime | extension was successfully linked.
[D 2022-03-31 15:53:10.578 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/home/bria/.jupyter/jupyter_notebook_config.json
[D 2022-03-31 15:53:10.578 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/etc/jupyter/jupyter_notebook_config.json
[D 2022-03-31 15:53:10.579 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2022-03-31 15:53:10.580 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/jupyter_resource_usage.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/jupyterlab_git.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/nbdime.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.json
[D 2022-03-31 15:53:10.584 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.json
[D 2022-03-31 15:53:10.584 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/home/bria/.local/etc/jupyter/jupyter_notebook_config.json
[D 2022-03-31 15:53:10.584 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/home/bria/.jupyter/jupyter_notebook_config.json
[D 2022-03-31 15:53:10.585 SingleUserLabApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/jupyter_resource_usage.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/jupyterlab_git.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.d/nbdime.json
/apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/etc/jupyter/jupyter_notebook_config.json
[I 2022-03-31 15:53:10.586 SingleUserLabApp manager:345] notebook_shim | extension was successfully linked.
[I 2022-03-31 15:53:10.637 SingleUserLabApp manager:367] notebook_shim | extension was successfully loaded.
[I 2022-03-31 15:53:10.638 SingleUserLabApp manager:367] jupyter_resource_usage | extension was successfully loaded.
[I 2022-03-31 15:53:10.639 SingleUserLabApp manager:367] jupyter_server_mathjax | extension was successfully loaded.
[I 2022-03-31 15:53:10.640 LabApp] JupyterLab extension loaded from /apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/lib/python3.9/site-packages/jupyterlab
[I 2022-03-31 15:53:10.640 LabApp] JupyterLab application directory is /apps/prod/easybuild/sl7.x86_64.foss-2021a/software/Python/3.9.5-GCCcore-10.3.0-jupyter/share/jupyter/lab
[I 2022-03-31 15:53:10.644 SingleUserLabApp manager:367] jupyterlab | extension was successfully loaded.
[I 2022-03-31 15:53:10.650 SingleUserLabApp manager:367] jupyterlab_git | extension was successfully loaded.
[I 2022-03-31 15:53:10.665 SingleUserLabApp manager:367] nbclassic | extension was successfully loaded.
[D 2022-03-31 15:53:10.839 SingleUserLabApp loader:499] Using default logger
[D 2022-03-31 15:53:10.840 SingleUserLabApp loader:499] Using default logger
[D 2022-03-31 15:53:10.840 SingleUserLabApp loader:499] Using default logger
[D 2022-03-31 15:53:10.840 SingleUserLabApp loader:499] Using default logger
[D 2022-03-31 15:53:10.840 SingleUserLabApp loader:499] Using default logger
[D 2022-03-31 15:53:10.840 SingleUserLabApp loader:499] Using default logger
[D 2022-03-31 15:53:10.840 SingleUserLabApp loader:499] Using default logger
[I 2022-03-31 15:53:10.843 SingleUserLabApp manager:367] nbdime | extension was successfully loaded.
[I 2022-03-31 15:53:10.843 SingleUserLabApp mixins:640] Starting jupyterhub-singleuser server version 2.2.2
[D 2022-03-31 15:53:10.848 SingleUserLabApp _version:74] jupyterhub and jupyterhub-singleuser both on version 2.2.2
[I 2022-03-31 15:53:10.848 SingleUserLabApp serverapp:2674] Serving notebooks from local directory: /home/bria
[I 2022-03-31 15:53:10.848 SingleUserLabApp serverapp:2674] Jupyter Server 1.15.6 is running at:
[I 2022-03-31 15:53:10.848 SingleUserLabApp serverapp:2674] http://computenode.eu.domain.com:59640/user/arnau.bria@domain.com/lab
[I 2022-03-31 15:53:10.849 SingleUserLabApp serverapp:2674] or http://127.0.0.1:59640/user/arnau.bria@domain.com/lab
[I 2022-03-31 15:53:10.849 SingleUserLabApp serverapp:2675] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2022-03-31 15:53:10.857 SingleUserLabApp mixins:596] Updating Hub with activity every 300 seconds
[D 2022-03-31 15:53:10.857 SingleUserLabApp mixins:558] Notifying Hub of activity 2022-03-31T13:53:10.600560Z
srun: Job step aborted: Waiting up to 12 seconds for job step to finish.
slurmstepd: error: *** STEP 14270408.0 ON computenode CANCELLED AT 2022-03-31T15:57:49 ***
slurmstepd: error: *** JOB 14270408 ON computenode CANCELLED AT 2022-03-31T15:57:49 ***
[C 2022-03-31 15:57:49.277 SingleUserLabApp serverapp:2124] received signal 15, stopping
[I 2022-03-31 15:57:49.302 SingleUserLabApp serverapp:2463] Shutting down 7 extensions

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.