linkedin / linkedin/pygradle

Multi platform pex distribution not supported

Open
#227 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
603
Forks
142
PR merge metrics
No merged PRs in 30d

Description

Eventhough we can set multiple platforms via pexOptions for pex builder, the wheels dependencies are not downloaded. This off course ends with an exception:
`11:24:40.539 [INFO] [org.gradle.process.internal.DefaultExecHandle] Starting process 'command '/workspace/tr/build/venv/bin/python''. Working directory: /workspace/tr Command: /workspace/tr/build/venv/bin/python /workspace/tr/build/venv/bin/pex --no-pypi --cache-dir /workspace/tr/build/pex-cache --output-file /workspace/tr/build/deployable/bin/test-runner --repo /build/wheel-cache --python-shebang /usr/bin/env python --entry-point tr.runner:main --platform=manylinux1-x86_64 --platform=macosx-10.6-x86_64 nose==1.3.7 pbr==3.1.1 pathtools==0.1.2 watchdog==0.8.3 PyYAML==3.12 lxml==4.1.1 pyserial==3.4 requests==2.11.1 Louie==1.1 test-runner==4.0.0 argh==0.26.2 psutil==5.4.3
11:24:40.539 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Environment for process 'command '/workspace/tr/build/venv/bin/python'': {PATH=/my.depot/main//Tools/3rdParty/MacOSX/itnl:/my.depot/main//Tools/3rdParty/MacOSX/ios-deploy:/my.depot/main//MacOSX/Build/cmake/3.8.1/bin:/my.depot/main//MacOSX/Build/ant/bin:/Library/Maven/apache-maven-3.3.9/bin:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS:/my.depot/main//MacOSX/Android/ndk9c:/my.depot/main//MacOSX/Android/sdk24/build-tools/23.0.1:/my.depot/main//MacOSX/Android/sdk24/platform-tools:/my.depot/main//MacOSX/Android/sdk24/tools, LC_MONETARY=en_US.UTF-8, LC_TIME=en_US.UTF-8, JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home, JAVA8_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home, LC_MESSAGES=en_US.UTF-8, TERM=xterm-256color, CT_DEVENV_HOME=/my.depot/main/, LANG=en_US.UTF-8, ANDROID_HOME=/my.depot/main//MacOSX/Android/sdk24, ANT_HOME=/my.depot/main//MacOSX/Build/ant, DISPLAY=/private/tmp/com.apple.launchd.wA0KSfYOF1/org.macosforge.xquartz:0, LC_COLLATE=en_US.UTF-8, LOGNAME=user, TERM_PROGRAM_VERSION=400, PWD=/workspace/tr, XPC_SERVICE_NAME=0, JAVA_MAIN_CLASS_31045=org.gradle.wrapper.GradleWrapperMain, NAVKIT_DATASET=/Volumes/Maps, SHELL=/bin/bash, TERM_PROGRAM=Apple_Terminal, JAVA7_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home, LSCOLORS=ExFxBxDxCxegedabagacad, CT_DEVENV_SYSTEM_HOME=/my.depot/main//MacOSX, ANDROID_NDK_HOME=/my.depot/main//MacOSX/Android/ndk9c, APP_ICON_31045=/workspace/tr/media/gradle.icns, OLDPWD=/workspace/tr, SECURITYSESSIONID=186a8, CLICOLOR=1, USER=user, TMPDIR=/var/folders/6t/zrvxmx8101l60k1w6l1f9pcw0000gn/T/, LC_NUMERIC=en_US.UTF-8, SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.pNNksY1bji/Listeners, NGL_DATASET=/Users/user/Perforce/user_mac_depot-ngl-dataset/, APP_NAME_31045=Gradle, XPC_FLAGS=0x0, TERM_SESSION_ID=FBE48B51-C6C9-44E2-8C26-0B4CDE74EED8, LC_ALL=, __CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0, Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.Xxicjt6IOD/Render, LC_CTYPE=en_US.UTF-8, HOME=/Users/user, SHLVL=1}
11:24:40.539 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: STARTING
11:24:40.539 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Waiting until process started: command '/workspace/tr/build/venv/bin/python'.
11:24:40.563 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: STARTED
11:24:40.563 [INFO] [org.gradle.process.internal.DefaultExecHandle] Successfully started process 'command '/workspace/tr/build/venv/bin/python''
11:24:40.563 [DEBUG] [org.gradle.process.internal.ExecHandleRunner] waiting until streams are handled...
11:24:41.373 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: FAILED
11:24:41.374 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Process 'command '/workspace/tr/build/venv/bin/python'' finished with exit value 1 (state: FAILED)
11:24:41.374 [LIFECYCLE] [com.linkedin.gradle.python.util.internal.pex.PexExecOutputParser] Could not satisfy all requirements for watchdog==0.8.3:
watchdog==0.8.3
`

Additional setup for building pex:
`buildPex.pexOptions` = ["--platform=manylinux1-x86_64", "--platform=macosx-10.6-x86_64"]`

Wheel cache dir:
`$ ls build/wheel-cache/
Louie-1.1-py2-none-any.whl lxml-4.1.1-cp27-cp27m-macosx_10_13_intel.whl pbr-3.1.1-py2.py3-none-any.whl requests-2.11.1-py2-none-any.whl
PyYAML-3.12-cp27-cp27m-macosx_10_13_intel.whl nose-1.3.7-py2-none-any.whl psutil-5.4.3-cp27-cp27m-macosx_10_13_intel.whl test_runner-4.0.0-py2-none-any.whl
argh-0.26.2-py2.py3-none-any.whl pathtools-0.1.2-py2-none-any.whl pyserial-3.4-py2.py3-none-any.whl watchdog-0.8.3-cp27-cp27m-macosx_10_13_intel.whl`

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the shown pex command with both --platform values and inspect the pex builder path that processes pexOptions and the build/wheel-cache directory. Done means dependencies such as watchdog are downloaded or selected for each requested platform and the multi-platform PEX build completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.