jupyterhub / jupyterhub/systemdspawner

A user servers's systemd service in a failed state fails to start up properly again

Open
#87 22 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
100
Forks
44
PR merge metrics
No merged PRs in 30d

Description

I'm having problems where if I leave a hub running for an extended period of time (weeks or longer) it becomes slow to respond and sometimes unable to start a user's server, for example:

```
May 27 03:07:25 littlehub python3[1910]: [I 2019-05-27 03:07:25.408 JupyterHub proxy:301] Checking routes
May 27 03:07:33 littlehub python3[1910]: [W 2019-05-27 03:07:33.210 JupyterHub base:714] User lukeedwards is slow to start (timeout=10)
May 27 03:07:33 littlehub python3[1910]: [I 2019-05-27 03:07:33.212 JupyterHub log:158] 202 POST /hub/api/users/lukeedwards/server (lachlan@106.69.254.220) 10064.93ms
May 27 03:08:02 littlehub python3[1910]: [I 2019-05-27 03:08:02.138 JupyterHub log:158] 200 GET /hub/admin (lachlan@106.69.254.220) 49.57ms
May 27 03:08:23 littlehub python3[1910]: [W 2019-05-27 03:08:23.294 JupyterHub user:471] lukeedwards's server failed to start in 60 seconds, giving up
May 27 03:08:23 littlehub python3[1910]: [E 2019-05-27 03:08:23.380 JupyterHub gen:593] Exception in Future .finish_user_spawn() done, defined at /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py:619> exception=TimeoutError('Timeout',)> after timeout
May 27 03:08:23 littlehub python3[1910]: Traceback (most recent call last):
May 27 03:08:23 littlehub python3[1910]: File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 589, in error_callback
May 27 03:08:23 littlehub python3[1910]: future.result()
May 27 03:08:23 littlehub python3[1910]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn
May 27 03:08:23 littlehub python3[1910]: await spawn_future
May 27 03:08:23 littlehub python3[1910]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/user.py", line 489, in spawn
May 27 03:08:23 littlehub python3[1910]: raise e
May 27 03:08:23 littlehub python3[1910]: File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/user.py", line 409, in spawn
May 27 03:08:23 littlehub python3[1910]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
May 27 03:08:23 littlehub python3[1910]: tornado.util.TimeoutError: Timeout
```

I've tried reloading the hub, and restarting the VM (Ubuntu 18.04 on Openstack) with no success.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.