jupyterlite / jupyterlite/pyodide-kernel
Support multiple `pyodide` kernelspecs
- Dominant language
- Python
- Stars
- 90
- Forks
- 47
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 11
Description
### Problem
At present, the server extension [registers](https://github.com/jupyterlite/pyodide-kernel/blob/v0.2.1/packages/pyodide-kernel-extension/src/index.ts#L55) a single kernelspec, and each kernel session is started with exactly that configuration.
In looking at supporting `pyodide-lock` (a la #49), a number of techniques currently in use, such as maintaining multiple versions of patched upstreams in the Warehouse `all.json`, will no longer work for the general case.
As a single `pyodide-lock.json` can only include one (or zero) version of each named package. The "interactive" solver will have less opportunity to load the "correct" package for corner cases, especially for a general purpose interactive site (as opposed to a well-engineered environment to complement some learning goals). Consider:
> _Open two notebooks, side-by-side, with two versions of the same package installed via_
> | notebook 1 | notebook 2 |
> |-|-|
> | `%pip install some-package=x.y` | `%pip install some-package=x.z` |
> | `import some_package` | `import some_package` |
> | `%timeit some_package.some_func()` | `%timeit some_package.some_func()` |
>
> _Compare the run times._
### Proposed Solution
Offer a way to more deeply configure the default `python` kernlspec, and create _extra_ named kernelspecs which could have separate:
- labels, e.g.
- `python` = `Python (Pyodide 0.25.0)`
- `python-custom` = `Python (Pyodide 0.25.0 + my-site)`
- launcher icons
- [`loadPyodide` configuration](https://github.com/pyodide/pyodide/blob/0.25.0/src/js/pyodide.ts#L61), e.g.
- environment variables
- `pyodide-lock.json` (or none at all, using the "stock" one provided with `pyodide.js`)
- python arguments
- can include `filterwarnings`, etc.
- packages to load
- but **only** if
- `pyodide-lock.json` is provided
- it is "complete" including `ipython` _and_ dependencies
### Additional context
#### For site users
- the `pyodide-lock.json` approach is probably a better UX
- multiple kernelspecs _will_ make things a bit noisier/more crowded
#### For site owners
- the `pyodide-lock.json` approach is strictly more reproducible
- unlike `all.json`, a `pyodide-lock.json` _can't_ reference PyPI-hosted dependencies, so a full solve is required
- there is no _lightweight_ way to get an authoritative `micropip.freeze` solve from a standard `CPython` build environment
- at present, the most reliable approach is likely to
- open a fresh browser
- `piplite.install` all packages on top of the baseline
- capture the output of a new `piplite.freeze`, as we may need to do some rewriting
- the CLI can probably only handle operating against the `python` kernelspec
- successive kernelspecs would likely need to be configured at build time through a new well-known location, e.g.
- `{lite_dir}/pyodide-kernelspecs/
- `{name}/`
- `kernelspec.json`
- the names of these would be injected into `jupyter-config-data`, and lazily resolved
#### Out of scope
- an even _more_ extreme approach could allow an `.ipynb` to offer its _own_ kernelspec overrides
- this is probably a bridge too far, but leaving a hook for this to be accomplished by an extension would be possible, especially if it worked for many kernels
- multiple kernelspecs _could_ in turn support multiple versions of `pyodide` on the same site
- this would increase maintenance burden, as each release would need to test against a matrix of supported versions instead of exactly one, losing robust typescript typing along the way, among other things
- the as-deployed structure would likely have to change from `/static/pyodide` to `/static/pyodide/{version-id}/
Contributor guide
Assessment
This issue has not been assessed yet.