apache / apache/arrow

[Python] cannot allocate memory in static TLS block exception raised when pyarrow is imported before _mysql

Open
#39,854 3 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The mysqlclient (MySQLdb) cannot be imported if pyarrow is imported first. An error is raised `ImportError: /lib64/libstdc++.so.6: cannot allocate memory in static TLS block`

I have experienced Red Hat and Oracle Linux distributions.

Steps to reproduce:

1. start docker container from oracle linux9 image `docker run -ti oraclelinux:9 bash`
2. install system dependencies
```
dnf install -y epel-release
dnf config-manager --set-enabled ol9_codeready_builder
dnf install gcc python3-devel python3-pip
```
The libmysql is 8.0.32 from oracle linux repositories but I had the same problem with I used libmysql from mysql
community repository .
3. install python dependencies (pyarrow and mysqlclient aka. MySQLdb)
```
pip3 install pyarrow==15.0.0 mysqlclient==2.2.1
```
4. mysql import is fine when pyarrow **is not** earlier imported
```
[root@930fd58ceb8a /]# python3 -c 'import MySQLdb; print("OK")'
OK
```
5. **fails** if pyarrow is imported before mysql
_ImportError: /lib64/libstdc++.so.6: cannot allocate memory in static TLS block_
```
[root@930fd58ceb8a /]# python3 -c 'import pyarrow; import MySQLdb; print("OK")'
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib64/python3.9/site-packages/MySQLdb/__init__.py", line 17, in
from . import _mysql
ImportError: /lib64/libstdc++.so.6: cannot allocate memory in static TLS block
```

I was probing the installed pyarrow library. The mysql import starts to fail after `pyarrow.lib` is imported https://github.com/apache/arrow/blob/maint-15.0.0/python/pyarrow/__init__.py#L65

When looking at shared libraries, `_mysql`, `pyarrow.lib` as we as `libmysqlclient` use the same `libstdc++.so.6 => /lib64/libstdc++.so.6`

```
[root@930fd58ceb8a /]# ldd /usr/local/lib64/python3.9/site-packages/MySQLdb/_mysql.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffda316c000)
libmysqlclient.so.21 => /usr/lib64/mysql/libmysqlclient.so.21 (0x00007f4c2d672000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4c2d469000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f4c2d392000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007f4c2d2ec000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f4c2ceb9000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4c2cea5000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f4c2cc7c000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f4c2cc60000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4c2dd11000)
libz.so.1 => /lib64/libz.so.1 (0x00007f4c2cc46000)
libm.so.6 => /lib64/libm.so.6 (0x00007f4c2cb6b000)
```

```
[root@930fd58ceb8a /]# ldd /usr/local/lib64/python3.9/site-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007fff96b91000)
libarrow_python.so => /usr/local/lib64/python3.9/site-packages/pyarrow/libarrow_python.so (0x00007f1798f6b000)
libarrow_dataset.so.1500 => /usr/local/lib64/python3.9/site-packages/pyarrow/libarrow_dataset.so.1500 (0x00007f1798da1000)
libparquet.so.1500 => /usr/local/lib64/python3.9/site-packages/pyarrow/libparquet.so.1500 (0x00007f1798538000)
libarrow_acero.so.1500 => /usr/local/lib64/python3.9/site-packages/pyarrow/libarrow_acero.so.1500 (0x00007f17983c5000)
libarrow.so.1500 => /usr/local/lib64/python3.9/site-packages/pyarrow/libarrow.so.1500 (0x00007f17957e0000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f17955b4000)
libm.so.6 => /lib64/libm.so.6 (0x00007f17954d9000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f17954bd000)
libc.so.6 => /lib64/libc.so.6 (0x00007f17952b4000)
librt.so.1 => /lib64/librt.so.1 (0x00007f17952af000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f17952aa000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f17952a3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f17994ed000)
```

```
[root@930fd58ceb8a /]# ldd /usr/lib64/mysql/libmysqlclient.so.21
linux-vdso.so.1 (0x00007ffe5cfaa000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007fb1976c3000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007fb19761d000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fb1971ea000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb1971d6000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb196faf000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb196f93000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb196d88000)
libz.so.1 => /lib64/libz.so.1 (0x00007fb196d6e000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb196c93000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb197e28000)
```

Interestingly importing other way round or setting the `LD_PRELOAD=/lib64/libstdc++.so.6` makes it work fine

```
[root@930fd58ceb8a /]# python3 -c 'import MySQLdb; import pyarrow; print("OK")'
OK

[root@930fd58ceb8a /]# LD_PRELOAD=/lib64/libstdc++.so.6 python3 -c 'import pyarrow; import MySQLdb; print("OK")'
OK
```

Could you help me understand what is going on and what would be the right fix for that?

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.