indygreg / indygreg/PyOxidizer
_sqlite3 module error when compiling for windows
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
Hi, this is an amazing project, the high level of engineering is noticeable all around.
I've built Linux and Mac executables and they seems to work well. However, on Windows 10, I'm getting an issue with one of the builtin extension modules, and I'd really appreciate some help.
The main issue is related to the sqlite extension module. I'm getting the following error both in Cmd and in WSL:
```
Traceback (most recent call last):
File "runpy", line 194, in _run_module_as_main
File "runpy", line 87, in _run_code
File "bin.q", line 43, in
File "sqlite3", line 23, in
File "sqlite3.dbapi2", line 27, in
ImportError: ('unable to load extension module library from memory', '_sqlite3')
```
I've tried multiple variations of the bzl configuration and couldn't get this to work properly.
In some of the variations I've managed to make it work when adding separate sqlite3 DLLs in another folder and adding it to the PATH. However, this misses the point of a simple installation flow for users, especially for Windows.
In other variations, Windows thinks that the executable is a virus and deletes it.
If I understand correctly, then I would need to make pyoxidizer extract the _sqlite3 shared library during startup as a filesystem-relative file, but i couldn't manage to do it, as it's not just a regular module, but seems like a semi-builtin one.
Any help or insights would be greatly supported.
Btw, the project i'm trying to oxidize is public and open-source, so i can provide github-action access if anyone would like to see the build details/logs (https://github.com/harelba/q/actions/runs/1347593026).
Hope this info helps, I'd be glad to test and provide more info as needed.
Current pyoxidizer.bzl content, which issues the error above (it's the best variation in terms of working all platforms, other ones created other issues in linux/mac):
```
# This file defines how PyOxidizer application building and packaging is
# performed. See PyOxidizer's documentation at
# https://pyoxidizer.readthedocs.io/en/stable/ for details of this
# configuration file format.
# Configuration files consist of functions which define build "targets."
# This function creates a Python executable and installs it in a destination
# directory.
def make_exe():
dist = default_python_distribution(python_version="3.8")
policy = dist.make_python_packaging_policy()
policy.set_resource_handling_mode("classify")
policy.resources_location = "in-memory"
policy.resources_location_fallback = "filesystem-relative:Lib"
policy.allow_in_memory_shared_library_loading = True
python_config = dist.make_python_interpreter_config()
python_config.run_module = "bin.q"
exe = dist.to_python_executable(
name="q",
packaging_policy=policy,
config=python_config,
)
exe.pip_install(["wheel"])
exe.add_python_resources(exe.pip_install(["-r", "requirements.txt"]))
exe.add_python_resources(exe.pip_install(["-e", "."]))
exe.add_python_resources(exe.read_package_root(
path="./",
packages=["bin"],
))
return exe
def make_embedded_resources(exe):
return exe.to_embedded_resources()
def make_install(exe):
# Create an object that represents our installed application file layout.
files = FileManifest()
# Add the generated executable to our install layout in the root directory.
files.add_python_resource(".", exe)
return files
def make_msi(exe):
# See the full docs for more. But this will convert your Python executable
# into a `WiXMSIBuilder` Starlark type, which will be converted to a Windows
# .msi installer when it is built.
builder = exe.to_wix_msi_builder(
# Simple identifier of your app.
"q",
# The name of your application.
"q-text-as-data",
# The version of your application.
"2.1.0",
# The author/manufacturer of your application.
"Harel Ben-Attia"
)
return builder
# Dynamically enable automatic code signing.
def register_code_signers():
# You will need to run with `pyoxidizer build --var ENABLE_CODE_SIGNING 1` for
# this if block to be evaluated.
if not VARS.get("ENABLE_CODE_SIGNING"):
return
# Use a code signing certificate in a .pfx/.p12 file, prompting the
# user for its path and password to open.
# pfx_path = prompt_input("path to code signing certificate file")
# pfx_password = prompt_password(
# "password for code signing certificate file",
# confirm = True
# )
# signer = code_signer_from_pfx_file(pfx_path, pfx_password)
# Use a code signing certificate in the Windows certificate store, specified
# by its SHA-1 thumbprint. (This allows you to use YubiKeys and other
# hardware tokens if they speak to the Windows certificate APIs.)
# sha1_thumbprint = prompt_input(
# "SHA-1 thumbprint of code signing certificate in Windows store"
# )
# signer = code_signer_from_windows_store_sha1_thumbprint(sha1_thumbprint)
# Choose a code signing certificate automatically from the Windows
# certificate store.
# signer = code_signer_from_windows_store_auto()
# Activate your signer so it gets called automatically.
# signer.activate()
# Call our function to set up automatic code signers.
register_code_signers()
# Tell PyOxidizer about the build targets defined above.
register_target("exe", make_exe)
register_target("resources", make_embedded_resources, depends=["exe"], default_build_script=True)
register_target("install", make_install, depends=["exe"], default=True)
register_target("msi_installer", make_msi, depends=["exe"])
# Resolve whatever targets the invoker of this configuration file is requesting
# be resolved.
resolve_targets()
```
requirements.txt file content:
```
six==1.11.0
flake8==3.6.0
setuptools<45.0.0
https://github.com/rogerbinns/apsw/releases/download/3.36.0-r1/apsw-3.36.0-cp38-cp38-win_amd64.whl
```
This requirements file is specific to windows, since i had another issue which is probably related. I was using sqlitebck module, but it doesn't have wheels for Windows, and this resulted in attempted build-from-source failures in Windows due to missing sqlite header files which I wasn't able to solve cleanly. I wouldn't mind going to apsw as part of solving this issue, as it provides similar functionality and is much more maintained, but the initial goal is to see if i can oxidize in Windows without any dependencies.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported ImportError for _sqlite3 and the resource-loading settings in pyoxidizer.bzl, then compare them with the Windows-specific requirements.txt configuration. Reproduce the build using the linked GitHub Actions run or an equivalent Windows environment. Done means the packaged executable loads sqlite3 on Windows without separately installed DLLs or an external PATH dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- build-system, devtools, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100