jupyterlite / jupyterlite/pyodide-kernel

Support multiple `pyodide` kernelspecs

Open
#83 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
90
Forks
47
Avg merge
3d 13h
Merged PRs (30d)
11

Description

### Problem

At present, the server extension [registers](https://github.com/jupyterlite/pyodide-kernel/blob/v0.2.1/packages/pyodide-kernel-extension/src/index.ts#L55) a single kernelspec, and each kernel session is started with exactly that configuration.

In looking at supporting `pyodide-lock` (a la #49), a number of techniques currently in use, such as maintaining multiple versions of patched upstreams in the Warehouse `all.json`, will no longer work for the general case.

As a single `pyodide-lock.json` can only include one (or zero) version of each named package. The "interactive" solver will have less opportunity to load the "correct" package for corner cases, especially for a general purpose interactive site (as opposed to a well-engineered environment to complement some learning goals). Consider:

> _Open two notebooks, side-by-side, with two versions of the same package installed via_
> | notebook 1 | notebook 2 |
> |-|-|
> | `%pip install some-package=x.y` | `%pip install some-package=x.z` |
> | `import some_package` | `import some_package` |
> | `%timeit some_package.some_func()` | `%timeit some_package.some_func()` |
>
> _Compare the run times._

### Proposed Solution

Offer a way to more deeply configure the default `python` kernlspec, and create _extra_ named kernelspecs which could have separate:

- labels, e.g.
- `python` = `Python (Pyodide 0.25.0)`
- `python-custom` = `Python (Pyodide 0.25.0 + my-site)`
- launcher icons
- [`loadPyodide` configuration](https://github.com/pyodide/pyodide/blob/0.25.0/src/js/pyodide.ts#L61), e.g.
- environment variables
- `pyodide-lock.json` (or none at all, using the "stock" one provided with `pyodide.js`)
- python arguments
- can include `filterwarnings`, etc.
- packages to load
- but **only** if
- `pyodide-lock.json` is provided
- it is "complete" including `ipython` _and_ dependencies

### Additional context

#### For site users

- the `pyodide-lock.json` approach is probably a better UX
- multiple kernelspecs _will_ make things a bit noisier/more crowded

#### For site owners

- the `pyodide-lock.json` approach is strictly more reproducible
- unlike `all.json`, a `pyodide-lock.json` _can't_ reference PyPI-hosted dependencies, so a full solve is required
- there is no _lightweight_ way to get an authoritative `micropip.freeze` solve from a standard `CPython` build environment
- at present, the most reliable approach is likely to
- open a fresh browser
- `piplite.install` all packages on top of the baseline
- capture the output of a new `piplite.freeze`, as we may need to do some rewriting
- the CLI can probably only handle operating against the `python` kernelspec
- successive kernelspecs would likely need to be configured at build time through a new well-known location, e.g.
- `{lite_dir}/pyodide-kernelspecs/
- `{name}/`
- `kernelspec.json`
- the names of these would be injected into `jupyter-config-data`, and lazily resolved

#### Out of scope

- an even _more_ extreme approach could allow an `.ipynb` to offer its _own_ kernelspec overrides
- this is probably a bridge too far, but leaving a hook for this to be accomplished by an extension would be possible, especially if it worked for many kernels
- multiple kernelspecs _could_ in turn support multiple versions of `pyodide` on the same site
- this would increase maintenance burden, as each release would need to test against a matrix of supported versions instead of exactly one, losing robust typescript typing along the way, among other things
- the as-deployed structure would likely have to change from `/static/pyodide` to `/static/pyodide/{version-id}/

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.