apache / apache/arrow

[Python] PyArrow package sets RUNPATH, making it incompatible with StaticX

Open
#43,538 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

### Description of the Issue
If a Python application includes PyArrow as a dependency then the application becomes incompatible with StaticX. StaticX fails, due to PyArrow using RUNPATH tags. Snippet of the error:
```
One or more libraries included in the PyInstaller archive uses unsupported RPATH/RUNPATH tags:
[truncated]
See https://github.com/JonathonReinhart/staticx/issues/188
```
That StaticX issue above directs to their more formal support documentation page: https://staticx.readthedocs.io/en/latest/rpath.html

### Why not remove StaticX?
To build and distribute relatively portable executable python applications, one of the more common and successful methods is to use PyInstaller to create an executable and then, for Linux targets, use StaticX to bundle the dynamic executable with their library dependencies so they can be run "anywhere" just like a static executable.

### Context Information
- As we only use StaticX for Linux targets, so far we have only tested and confirmed this issue with pyarrow Linux packages and the following:
- Version of Python: 3.11
- Platform: Ubuntu 20.04.6 LTS
- Version of PyInstaller: 6.9.0 (Latest)
- Version of StaticX: 0.14.1 (Latest)
- Version of PyArrow: 17.0.0 (Latest); 13.0.0; We presume the in-between versions are also affected.
- Version of PyArrow Without Error: 12.0.1; 11.0.0; We didn't test anything lower.

### Minimal Example
```
import pyarrow

if __name__ == '__main__':
print("simple test")
```
On a Linux machine install the "pyinstaller", "staticx", and "pyarrow" packages in your virtual environment. Then build the script with PyInstaller (E.g.: pyinstaller simple.py --name simple --onefile). Then attempt to use StaticX to bundle the executable (E.g.: staticx dist/simple dist/simplex). StaticX will fail with the following error:

### Full Error Message
```
staticx: Unsupported PyInstaller input

One or more libraries included in the PyInstaller archive uses unsupported RPATH/RUNPATH tags:

/tmp/staticx-pyi-lo4_jyb_/pyarrow/_acero.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_azurefs.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_compute.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_csv.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_dataset.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_dataset_orc.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_dataset_parquet.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_dataset_parquet_encryption.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_feather.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_flight.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_fs.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_gcsfs.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_hdfs.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_json.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_orc.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_parquet.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_parquet_encryption.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_s3fs.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/_substrait.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/lib.cpython-311-x86_64-linux-gnu.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow.so.1700: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow_acero.so.1700: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow_dataset.so.1700: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow_flight.so.1700: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow_python.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow_python_flight.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow_python_parquet_encryption.so: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libarrow_substrait.so.1700: DT_RUNPATH='$ORIGIN'
/tmp/staticx-pyi-lo4_jyb_/pyarrow/libparquet.so.1700: DT_RUNPATH='$ORIGIN'

See https://github.com/JonathonReinhart/staticx/issues/188
```

### Potential Resolution
I won't pretend to understand the various complexities of your build scripts. But it seems like PyArrow would need to either not set RPATH/RUNPATH at all, or set RPATH rather than RUNPATH (since StaticX does support RPATH relative to $ORIGIN).

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.