Canonical way to create a kernel when creating a new conda environment
- Dominant language
- Jupyter Notebook
- Stars
- 13.3k
- Forks
- 5.8k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 7
Description
I'm looking to see if there is a more standard, "canonical", way to create a kernel when creating a conda environment. It seems to be a few answers that abound, but no standard. This would really help for those that teach classes or create tutorials.
As a concrete example, I will go through what happens on my mac. Let's say I create a conda environment with Python and Jupyter installed
$ conda create -n test_env python jupyter
This installs ipykernel as a dependency. I then activate the environment and launch a jupyter notebook
$ conda activate test_env
$ jupyter notebook
When the server is started, creating a notebook by clicking the 'New' button reveals a dropdown menu with 'Python 3' as the only choice (let's assume there are no other environments). Launching a new notebook from here will not execute Python from the test_env but instead from the base environment.
This seems counterintuitive, but I can get a list of the kernels to get more info with the following:
$ jupyter kernelspec list
python3 /Users/Ted/Library/Jupyter/kernels/python3
Reading the docs on the [kernel specs
](https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs), I see that this is part of the User path which takes precedence over the others. It appears that Env is next, followed by System.
This means that no matter what environment I am in, the base environment will take precedence for the 'python3' kernel. Now, if I look at the location where the Env kernel is, I see that a 'python3' kernel is also listed there.
$ ls /Users/Ted/anaconda3/envs/test_env/share/jupyter/kernels
python3
The most popular solution at this point appears to be to manually create a kernel with:
$ python -m ipykernel install --user --name test_env --display-name "Python (test_env)"
So, this works, and now I can access my test_env Python installation by starting a notebook with this kernel. But, it appears this is rather messy and an alternative (better?) solution would be to have no User kernels at all.
From the first response of this [GitHub issue](https://github.com/jupyter/notebook/issues/2898), I can just remove the User 'python3' kernel. As far as I know, this is the only documented place where this advice is given. So I do this with:
$ jupyter kernelspec remove python3
Now, this kernel is removed so when running the command `jupyter notebook`, it will find the 'python3' kernel for the current environment as it is next in line. Creating a new notebook now executes Python from the installation in the active environment. Moreso, any new environment will automatically be set up so that the 'python3' kernel is pointing to its environments location.
### The User 'python3' kernel does not always exist
A student of mine told me that he had no User 'python3' kernel so launching his jupyter notebooks after environment creation automatically started in the 'correct' place. Apparently this is the case for some users.
*If anyone could explain why some people have User 'python3' kernels and others don't that would be great.*
### More questions
I know this is a long post, but I feel there is lots of confusion surrounding conda environment creation and kernels.
* It seems strange to even have the opportunity to run a different kernel other than the one created specifically for the environment where jupyter is launched from. Is there a purpose for this?
* To me it would appear much more intuitive that when you begin a jupyter notebook within an environment, that environment's kernel is prioritized. Shouldn't the environment have higher priority?
* Which solution could be more canonical? Running `python -m ipykernel install --user --name test_env` or `jupyter kernelspec remove python3`?
* Is it possible that I created the User 'python3' kernel unknowingly? It does seem like many people have issues with this.
I made a similar post on [Stack Overflow](https://stackoverflow.com/questions/39604271/conda-environments-not-showing-up-in-jupyter-notebook/54985829#54985829) detailing my solution.
Contributor guide
Research direction
Begin with the kernel-spec discovery and precedence documentation linked in the report, then reproduce the listed conda, jupyter, and kernelspec commands. Compare the User and environment python3 entries and trace how the active notebook selects one. Done would require a decided canonical workflow or behavior, with the source of User kernels and the expected environment priority documented or tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter, jupyter-notebook, python
- Domain
- developer-experience, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100