MagicStack / MagicStack/httptools

Custom build_ext breaks parallel build/wheel

Open
#126 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
107
PR merge metrics
No merged PRs in 30d

Description

Our build system typically uses setup.py to generate a wheel via bdist_wheel but to speed up compilation of extensions we also explictly call build and pass -j:

python3 ./setup.py build  -j 8 bdist_wheel

This works for 99% of modules but fails with httptools:


running build
running build_py
creating build/lib.linux-aarch64-cpython-312/httptools
copying httptools/_version.py -> build/lib.linux-aarch64-cpython-312/httptools
copying httptools/__init__.py -> build/lib.linux-aarch64-cpython-312/httptools
creating build/lib.linux-aarch64-cpython-312/httptools/parser
copying httptools/parser/errors.py -> build/lib.linux-aarch64-cpython-312/httptools/parser
copying httptools/parser/__init__.py -> build/lib.linux-aarch64-cpython-312/httptools/parser
running egg_info
writing httptools.egg-info/PKG-INFO
writing dependency_links to httptools.egg-info/dependency_links.txt
writing requirements to httptools.egg-info/requires.txt
writing top-level names to httptools.egg-info/top_level.txt
/usr/lib/python3/dist-packages/pbr/git.py:28: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
reading manifest file 'httptools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'httptools.egg-info/SOURCES.txt'
copying httptools/parser/cparser.pxd -> build/lib.linux-aarch64-cpython-312/httptools/parser
copying httptools/parser/parser.pyx -> build/lib.linux-aarch64-cpython-312/httptools/parser
copying httptools/parser/python.pxd -> build/lib.linux-aarch64-cpython-312/httptools/parser
copying httptools/parser/url_cparser.pxd -> build/lib.linux-aarch64-cpython-312/httptools/parser
copying httptools/parser/url_parser.pyx -> build/lib.linux-aarch64-cpython-312/httptools/parser
running build_ext
building 'httptools.parser.parser' extension
creating build/temp.linux-aarch64-cpython-312/httptools/parser
building 'httptools.parser.url_parser' extension
creating build/temp.linux-aarch64-cpython-312/vendor/llhttp/src
creating build/temp.linux-aarch64-cpython-312/vendor/http-parser
aarch64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/include -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/src -I/usr/include/python3.12 -c httptools/parser/parser.c -o build/temp.linux-aarch64-cpython-312/httptools/parser/parser.o -O2
aarch64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/http-parser -I/usr/include/python3.12 -c httptools/parser/url_parser.c -o build/temp.linux-aarch64-cpython-312/httptools/parser/url_parser.o -O2
aarch64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/http-parser -I/usr/include/python3.12 -c vendor/http-parser/http_parser.c -o build/temp.linux-aarch64-cpython-312/vendor/http-parser/http_parser.o -O2
aarch64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-aarch64-cpython-312/httptools/parser/url_parser.o build/temp.linux-aarch64-cpython-312/vendor/http-parser/http_parser.o -L/usr/lib/aarch64-linux-gnu -o build/lib.linux-aarch64-cpython-312/httptools/parser/url_parser.cpython-312-aarch64-linux-gnu.so
aarch64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/include -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/src -I/usr/include/python3.12 -c vendor/llhttp/src/api.c -o build/temp.linux-aarch64-cpython-312/vendor/llhttp/src/api.o -O2
aarch64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/include -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/src -I/usr/include/python3.12 -c vendor/llhttp/src/http.c -o build/temp.linux-aarch64-cpython-312/vendor/llhttp/src/http.o -O2
aarch64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/include -I/work/ross/build/tmp/work/core2-64-poky-linux/python3-httptools/0.6.4/sources/httptools-0.6.4/vendor/llhttp/src -I/usr/include/python3.12 -c vendor/llhttp/src/llhttp.c -o build/temp.linux-aarch64-cpython-312/vendor/llhttp/src/llhttp.o -O2
aarch64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-aarch64-cpython-312/httptools/parser/parser.o build/temp.linux-aarch64-cpython-312/vendor/llhttp/src/api.o build/temp.linux-aarch64-cpython-312/vendor/llhttp/src/http.o build/temp.linux-aarch64-cpython-312/vendor/llhttp/src/llhttp.o -L/usr/lib/aarch64-linux-gnu -o build/lib.linux-aarch64-cpython-312/httptools/parser/parser.cpython-312-aarch64-linux-gnu.so

running bdist_wheel
installing to build/bdist.linux-aarch64/wheel
running install
running install_lib
running build_ext
error: don't know how to compile C/C++ code on platform 'posix' with '<distutils.compilers.C.unix.Compiler object at 0xee4af1454c80>' compiler

This behaviour isn't specific to the use of -j, simply doing python3 setup.py build bdist_wheel results in the build failing.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with setup.py and reproduce the failure using python3 setup.py build bdist_wheel, then compare the initial build_ext run with the build_ext invocation during bdist_wheel. The work is done when this command completes successfully and produces a wheel without the compiler error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.