jupyterhub / jupyterhub/batchspawner

"Remote port selection" allows redesigning batchspawner to avoid polling/spawner interaction

Open
#146 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
207
Forks
136
PR merge metrics
No merged PRs in 30d

Description

With the introduction of "select the port on the remote side and have it convey it back to us" (#58, #141), I realized we have fundamentally changed how batchspawner works. Previously, we polled the batch system in order to wait until things started and set the hostname/port. Now, we poll the batch system to get hostname, but wait for port from the remote side.

What if the remote side also sent the hostname in via the API? If that happens, then we have no need to poll batch system at all. In fact, there is hardly any need to do much integration with each batch system - batchspawer could become a "just start some command, it runs wherever and wait for it to send info on how it ran back to us". The only extra command we'd have to be able to run is "stop". Taken to the extreme, we woludn't even have to capture the job id because that could be sent (but would leave us high chance of orphan jobs, so we probably wouldn't want to do that).

But then what does batchspawner become? Really then it doesn't need to know much about batch systems, it's similar to ssh spawner, kubernetes, whatever (though now every server knows its external address!). The "report back" function could be made a part of the general notebook server - but wait, it's not so easy because it means the remote side has to integrate with the batchspawner or execution environment instead of server side, so (for example) in our case each local side also needs batchspawner installed (which happens to be now the case, which is a subtle but important thing to keep in mind).

Thoughts? (Mine is it's a clever idea, but not something I'm planning on using myself, but an "alive" API endpoint that does a bit more could simplify and unify some spawners.)

cc @minrk since this gets into the philosophy of jupyterhub.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.