Don't call pip freeze to check package version (performance cost in general, permission problems in specific case)

Start by locating the python::pip implementation that runs pip freeze | grep during package checks, then compare the behavior of pip list, pip show, and direct pip install using the examples in the issue. Done means package presence and version checks no longer incur the unrelated pip freeze repository scans or permission failures, with coverage for the supported version formats.

Summary

I faced permission problems which in the end turned out to be caused by python::pip calling pip freeze calling hg showconfig on modules unrelated to the installation in charge. While problems as such are specific to my somewhat specific way of using puppet, they can be easily resolved, and fixes make things working much faster in the general case.

Appetizer

$ time pip freeze | grep -e '^SQLAlchemy'
SQLAlchemy==0.8.4
(…)
real    0m5.488s

time pip show SQLAlchemy
(…)
Version: 0.8.4
(…)
real    0m0.198s

Permission problem

I use some Puppet manifest to support managing my development environment. It's light case, without any puppet master, with Puppet serving as a helper just to ensure various packages are installed. So I do things like

sudo puppet apply manifests/devel.pp --modulepath=./modules

I agree that this is not perfectly safe way of doing things, but that's also not a production…

With python::pip installed packages it used to work ……… until I performed some manual pip install --user --edit (on completely unrelated modules). Since then, Puppet manifests started to ignore python::pip modules, and debugging output showed up info like:

Debug: Exec[pip_install_cram](provider=posix): Executing check 'pip freeze | grep -i -e ^cram=='
Debug: Executing 'pip freeze | grep -i -e ^cram=='
(…)
Debug: /Stage[main]/Python_local/Python::Pip[cram]/Exec[pip_install_cram]/unless: not trusting file /home/marcink/DEV_hg/mercurial/extension_utils/.hg/hgrc from untrusted user marcink, group marcink
(…)
Debug: /Stage[main]/Python_local/Python::Pip[cram]/Exec[pip_install_cram]/unless: Error when trying to get requirement for VCS system Command /usr/bin/hg showconfig paths.default failed with error code 1 in /home/marcink/DEV_hg/mercurial/extension_utils, falling back to uneditable format
Debug: /Stage[main]/Python_local/Python::Pip[cram]/Exec[pip_install_cram]/unless: Could not determine repository location of /home/marcink/DEV_hg/mercurial/extension_utils
(… similar things repeated for all locally installed repositories …)
Debug: /Stage[main]/Python_local/Python::Pip[cram]/Exec[pip_install_cram]/unless: cram==0.6

(here I am python::pip-ing cram which is in no way related to locally installed extension_utils)

In the end it turns out that things works so:

• python::pip uses pip freeze | grep Package (or | grep Package==X.Y.Z) to check whether package is installed
• pip freeze scans all locally installed packages (and that's my fault that it scans packages installed on my local account, as sudo does not clear environment well enough, and I wouldn't open this ticket unless there were other reasons to avoid calling it)
• as pip freeze intends to emit exact specification of how to reinstall any and all of those packages, for every one of them it looks for best way of how to specify the dependency
• in particular for packages installed „for edit" it applies some heuristics by looking at whether source directory is some repository, and if so, where it was pulled from, in the end emitting specs like -e hg+ssh://master/mercurial/extension_utils@f90a7710d6b2cba2cce9dbe61e5af9f7a33e52c3#egg=mercurial_extension_utils-default
• to get the latter info it spawns hg (or git if it were git repo) to ask it for various details (aforementioned hg showconfig paths.default prints default repository remote, which in typical case is the url repo was cloned from), and this commands fails due to my sudo
• leaving apart my permission issues, this whole bunch of calls is simply completely unnecessary, costly, and unrelated to the task, after all the only thing which needs to be checked is whether installed module (here: cram) is installed and in which version.

Suggested fix

1. Simpler but less effective version: swapping pip freeze to pip list and patching $grep_regexp to accomodate different syntax (pip list emits things like cram (0.6) instead of cram==0.6 emitted by pip freeze resolves my permission issue but does not offer noticeable performance gains (btw, for locally installed dirs the output is much more useful, for example mercurial-extension-utils (1.0.0, /home/marcink/DEV_hg/mercurial/extension_utils)).
   To get that fix I simply replaced freeze with list and edited the regexp this way

   if $ensure =~ /^((19|20)[0-9][0-9]-(0[1-9]|1[1-2])-([0-2][1-9]|3[0-1])|[0-9]+\.[0-9]+(\.[0-9]+)?)$/ {
     $grep_regex =  "^" + $pkgname  + " [\\(]" + $ensure + '[\\)]$'
   } else {
     $grep_regex = "^${pkgname} "
   }
   

2. Much better idea would be to wrap pip show PKGNAME instead, see appetizer above for explanation why.

3. I am not 100% sure whether whole unless is needed at all. Blindly calling pip install PkgName==Version may be faster than checking whether this call is needed. For example, when package cram is alredy installed at version 0.6, I have:

$ time pip freeze | grep cram
cram==0.6
real    0m5.535s

$ time pip show cram

---
Name: cram
Version: 0.6
(…)

real    0m0.199s
(…)

$ time pip install cram==0.6
Requirement already satisfied (use --upgrade to upgrade): cram==0.6 in
(…)
real    0m0.197s

so, as we can see the actual cost of checking whether pip install is needed is exactly equal to just calling it.