PyO3 / PyO3/pyo3

Interpreter is not found when using Anaconda

Open
#1,554 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dependencies
Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

🐛 Bug Reports

When reporting a bug, please provide the following information. If this is not a bug report you can just discard this template.

🌍 Environment
  • Your operating system and version: Windows 10
  • Your python version: 3.8.5
  • How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: Anaconda default Python installation
  • Your Rust version (rustc --version): 1.53.0-nightly
  • Your PyO3 version: 0.13.2
  • Have you tried using latest PyO3 main (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: Yes
💥 Reproducing

Please provide a minimal working example. This means both the Rust code and the Python.
The following code compiles, but crashes when using Anaconda distribution (using the default installation).

use pyo3::prelude::*;

fn main() -> PyResult<()>{
    let gil = Python::acquire_gil();
    let py = gil.python();

    let result = py.run("print('it works')", None, None);

    if let Err(ref err) = result {
        println!("{:?}", err);
    }
    Ok(())
}

The output I got:

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\bug\\target\\debug\\bug.exe'
  sys.base_prefix = 'C:\\Users\\balog\\anaconda3'
  sys.base_exec_prefix = 'C:\\Users\\balog\\anaconda3'
  sys.executable = 'C:\\bug\\target\\debug\\bug.exe'
  sys.prefix = 'C:\\Users\\balog\\anaconda3'
  sys.exec_prefix = 'C:\\Users\\balog\\anaconda3'
  sys.path = [
    'C:\\Users\\balog\\anaconda3\\python38.zip',
    '.\\DLLs',
    '.\\lib',
    'C:\\bug\\target\\debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000e6c (most recent call first):
<no Python frame>
error: process didn't exit successfully: `target\debug\bug.exe` (exit code: 1)

The sys executables are pointing to wrong locations, so it seems it couldn't find the Python interpreter. I haven't checked how PyO3 tries to find the interpreter internally, but as far as I know Anaconda doesn't set PYTHONPATH nor PYTHONHOME env variables and that might be the cause that breaks it. Anaconda sets up the CONDA_PREFIX, which can be - at least in my case - set as PYTHONHOME, so it's my temporal workaround:

use pyo3::ffi;
use pyo3::prelude::*;
use widestring::WideCString;

fn main() -> PyResult<()> {
    if let Some(PYTHONHOME) = std::env::var_os("CONDA_PREFIX") {
        unsafe {
            ffi::Py_SetPythonHome(
                WideCString::from_str(PYTHONHOME.to_str().unwrap())
                    .unwrap()
                    .as_ptr(),
            );
        }
    }

    let gil = Python::acquire_gil();
    let py = gil.python();

    let result = py.run("print('it works')", None, None);

    if let Err(ref err) = result {
        println!("{:?}", err);
    }
    Ok(())
}

However I'm not sure it works out-of-the-box.

The output of conda info -a:


     active environment : base
    active env location : C:\Users\balog\anaconda3
            shell level : 1
       user config file : C:\Users\balog\.condarc
 populated config files : 
          conda version : 4.9.2
    conda-build version : 3.20.5
         python version : 3.8.5.final.0
       virtual packages : __cuda=9.1=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\balog\anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\balog\anaconda3\pkgs
                          C:\Users\balog\.conda\pkgs
                          C:\Users\balog\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\balog\anaconda3\envs
                          C:\Users\balog\.conda\envs
                          C:\Users\balog\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Windows/10 Windows/10.0.19041
          administrator : False
             netrc file : None
           offline mode : False

# conda environments:
#
base                  *  C:\Users\balog\anaconda3

sys.version: 3.8.5 (default, Sep  3 2020, 21:29:08) [...
sys.prefix: C:\Users\balog\anaconda3
sys.executable: C:\Users\balog\anaconda3\python.exe
conda location: C:\Users\balog\anaconda3\lib\site-packages\conda
conda-build: C:\Users\balog\anaconda3\Scripts\conda-build.exe
conda-convert: C:\Users\balog\anaconda3\Scripts\conda-convert.exe
conda-debug: C:\Users\balog\anaconda3\Scripts\conda-debug.exe
conda-develop: C:\Users\balog\anaconda3\Scripts\conda-develop.exe
conda-env: C:\Users\balog\anaconda3\Scripts\conda-env.exe
conda-index: C:\Users\balog\anaconda3\Scripts\conda-index.exe
conda-inspect: C:\Users\balog\anaconda3\Scripts\conda-inspect.exe
conda-metapackage: C:\Users\balog\anaconda3\Scripts\conda-metapackage.exe
conda-render: C:\Users\balog\anaconda3\Scripts\conda-render.exe
conda-server: C:\Users\balog\anaconda3\Scripts\conda-server.exe
conda-skeleton: C:\Users\balog\anaconda3\Scripts\conda-skeleton.exe
conda-verify: C:\Users\balog\anaconda3\Scripts\conda-verify.exe
user site dirs: 

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: base
CONDA_EXE: C:\Users\balog\anaconda3\condabin\..\Scripts\conda.exe
CONDA_EXES: "C:\Users\balog\anaconda3\condabin\..\Scripts\conda.exe"  
CONDA_PREFIX: C:\Users\balog\anaconda3
CONDA_PROMPT_MODIFIER: (base) 
CONDA_PYTHON_EXE: C:\Users\balog\anaconda3\python.exe
CONDA_ROOT: C:\Users\balog\anaconda3
CONDA_SHLVL: 1
CURL_CA_BUNDLE: <not set>
HOMEPATH: \Users\balog
PATH: C:\Users\balog\anaconda3;C:\Users\balog\anaconda3\Library\mingw-w64\bin;C:\Users\balog\anaconda3\Library\usr\bin;C:\Users\balog\anaconda3\Library\bin;C:\Users\balog\anaconda3\Scripts;C:\Users\balog\anaconda3\bin;C:\Users\balog\anaconda3;C:\Users\balog\anaconda3\Library\mingw-w64\bin;C:\Users\balog\anaconda3\Library\usr\bin;C:\Users\balog\anaconda3\Library\bin;C:\Users\balog\anaconda3\Scripts;C:\Users\balog\anaconda3\bin;C:\Users\balog\anaconda3\condabin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\pysprint-cli\bin;C:\Users\balog\.cargo\bin;C:\Users\balog\AppData\Local\Microsoft\WindowsApps;C:\Users\balog\anaconda3\python.exe;.
PSMODULEPATH: C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the minimal Rust example on Windows with the reported Anaconda installation, then inspect PyO3's Python interpreter initialization and path configuration. Done means the example runs with Anaconda without requiring the manual CONDA_PREFIX/PYTHONHOME workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
anaconda, python, rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.