Interoperability with pip

Summary

This is a suggestion to improve interoperability between Spack environments and pip. Essentially I propose to make pip install packages in the activated Spack environment, instead of the python prefix.

Rationale

Pip should not pollute the python installation prefix.

Description

The idea is to enable this workflow:

spack create myenv
spack activate myenv
spack install py-pip
pip install rich
python -m rich  # works
despacktivate
python -m rich  # expect ImportError

At the moment, the workflow above sort of works, but the pip installation pollutes the system python installation, so after the despacktivate the package is still available and may conflict with other environments.

Additional information

I understand that interoperability with another package manager is tricky, but I believe pip is so widely used that it is worth the effort. Note that conda supports this as well with some caveats.

My suggestion for an implementation is this:

• On spack env activate configure pip to install packages in a directory of the environment, say myenv/.spack-env/pip/. At the command line this can be done with pip install --prefix, and there’s surely a way to set the prefix more permanently. Edit: environment variables like VIRTUAL_ENV should do it, or PYTHONUSERBASE together with --user.
• Add the pip installation prefix configured above to the PYTHONPATH that is loaded with the spack environment.
• Now Spack-installed python packages get symlinked into the environment view, and pip-installed packages are picked up from the permanent location in the environment directory.

There are surely some pitfalls, for instance:

• When dependencies of the pip-installed packages are provided by spack but then get removed from the view. I think a pip reinstall of the package by the user would fix this, and there is little else to do about it from the Spack side.
• Priorities: Should the Spack or pip packages come first in the PYTHONPATH? I believe it should be the Spack packages, so a pip reinstall of a package would pick up updated Spack packages.
• See link above for docs on how conda deals with these issues.

General information

• I have run spack --version and reported the version of Spack
• I have searched the issues of this repo and believe this is not a duplicate

Start with the spack env activate workflow described in the issue and compare pip behavior before and after activation. Done means pip-installed packages are available inside the active Spack environment, do not pollute the Python prefix, and are unavailable after deactivation; resolve the proposed prefix and PYTHONPATH behavior, including package priority.