indygreg / indygreg/PyOxidizer

Windows: "unable to load extension module library from memory" for sqlite3

Open
#322 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

On Windows 10 (version 19042.572) with PyOxidizer 0.9.0 (rustc 1.47.0 (18bf6b4f0 2020-10-07) stable-x86_64-pc-windows-msvc), place the following in `pyoxidizer.bzl` in an otherwise empty directory.

```bzl
def make():
dist = default_python_distribution(flavor="standalone_dynamic")
python_config = dist.make_python_interpreter_config()
python_config.config_profile = "python"
python_config.run_mode = "eval:import sqlite3"

policy = dist.make_python_packaging_policy()
policy.set_resource_handling_mode("classify")
policy.resources_location = "in-memory"
policy.allow_in_memory_shared_library_loading = True

exe = dist.to_python_executable(
name="pyapp",
packaging_policy=policy,
config=python_config,
)

files = FileManifest()
files.add_python_resource(".", exe)
return files

register_target("install", make, default=True)
resolve_targets()
```

Executing `pyoxidizer run` gives the following output (build output omitted):
```python-traceback
Traceback (most recent call last):
File "", line 1, in
File "sqlite3", line 23, in
File "sqlite3.dbapi2", line 27, in
ImportError: ('unable to load extension module library from memory', '_sqlite3')
```

The error goes away if you make the following change to `pyoxidizer.bzl`:

```diff
- policy.allow_in_memory_shared_library_loading = True
+ policy.resources_location_fallback = "filesystem-relative:Lib"
+ policy.allow_in_memory_shared_library_loading = False
```

Granted, the docs for [`PythonPackagingPolicy.allow_in_memory_shared_library_loading`](https://pyoxidizer.readthedocs.io/en/stable/config_type_python_packaging_policy.html#config-type-python-packaging-policy-allow-in-memory-shared-library-loading) do say (emphasis mine):

> Some platforms (notably Windows) allow opening shared libraries from a memory address. This mode of opening shared libraries allows libraries to be embedded in binaries without having to statically link them. However, **not every library works correctly when loaded this way.**

So I can't tell if this error is documented behavior or not. At the very least it would be helpful for the documentation to say how to predict what will work and what won't. But it'd also be nice if it worked...

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Windows failure using the minimal pyoxidizer.bzl configuration and run pyoxidizer run with sqlite3 imported. Compare it with the filesystem-relative fallback, then determine whether the in-memory loading behavior needs a fix or clearer documentation describing sqlite3 compatibility. Done means the failure is resolved or the documented limitation and workaround are precise.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, sqlite
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.