[VL] Gluten on Spark 3.2 and even in newer spark with latest throwing JVM fatal error

### Backend

VL (Velox)

### Bug description

[Expected to run with JDK 8 and Ubuntu OS for gluten 0.5.0 here is the spark config used for same
`conf = {
"spark.plugins":"io.glutenproject.GlutenPlugin",
"spark.shuffle.manager":"org.apache.spark.shuffle.sort.ColumnarShuffleManager",
"spark.memory.offHeap.enabled": "true",
"spark.memory.offHeap.size":"1G",
"spark.submit.deployMode":"client",
"spark.gluten.loadLibFromJar":"true",
"spark.jars":"https://github.com/apache/incubator-gluten/releases/download/0.5.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04-0.5.0-SNAPSHOT.jar,https://github.com/apache/incubator-gluten/releases/download/0.5.0/gluten-thirdparty-lib-ubuntu-20.04.jar"
}`
] and [ Stack Trace
```
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00007f02e4e5eca1, pid=720, tid=0x00007f03acfc9700
#
# JRE version: OpenJDK Runtime Environment (8.0_342-b07) (build 1.8.0_342-8u342-b07-0ubuntu1~20.04-b07)
# Java VM: OpenJDK 64-Bit Server VM (25.342-b07 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libgluten.so+0x469ca1] gluten::SetBackendFactory(std::function ()>)+0x21
#
# Core dump written. Default location: /home/ray/fsx/workspace/ravi.rana@dream11.com/spark350/spark350/velox/core or core.720
#
# An error report file with more information is saved as:
# /home/ray/fsx/workspace/ravi.rana@dream11.com/spark350/spark350/velox/hs_err_pid720.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.`
```
].

OS : Ubuntu 20.04

Jars of gluten used :
1. https://github.com/apache/incubator-gluten/releases/download/0.5.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04-0.5.0-SNAPSHOT.jar
2. https://github.com/apache/incubator-gluten/releases/download/0.5.0/gluten-thirdparty-lib-ubuntu-20.04.jar

CPU : AMD EPYC 7571
CPU arch : x86_64 GNU/Linux

Added thread dump for same as well
[hs_err_pid720.log](https://github.com/user-attachments/files/16707582/hs_err_pid720.log)

### Spark version

Spark-3.2.x

### Spark configurations

{
"spark.plugins":"io.glutenproject.GlutenPlugin",
"spark.shuffle.manager":"org.apache.spark.shuffle.sort.ColumnarShuffleManager",
"spark.memory.offHeap.enabled": "true",
"spark.memory.offHeap.size":"1G",
"spark.submit.deployMode":"client",
"spark.gluten.loadLibFromJar":"true",
"spark.jars":"https://github.com/apache/incubator-gluten/releases/download/0.5.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04-0.5.0-SNAPSHOT.jar,https://github.com/apache/incubator-gluten/releases/download/0.5.0/gluten-thirdparty-lib-ubuntu-20.04.jar"
}

### System information

Velox System Info v0.0.2
Commit: 07c1ce7b42c711ab9bc2a7d654a41bb7454bc4d1
CMake Version: 3.16.3
System: Linux-5.10.215-203.850.amzn2.x86_64
Arch: x86_64
CPU Name: Model name: AMD EPYC 7571
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 9.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 9.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt

Conda Env

```
# packages in environment at /home/ray/anaconda3:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
absl-py 1.3.0 pypi_0 pypi
adal 1.2.7 pypi_0 pypi
aiofiles 22.1.0 pypi_0 pypi
aiohttp 3.8.6 pypi_0 pypi
aiohttp-cors 0.7.0 pypi_0 pypi
aiorwlock 1.3.0 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
aiosqlite 0.19.0 pypi_0 pypi
ansiwrap 0.8.4 pypi_0 pypi
anyio 4.2.0 pypi_0 pypi
applicationinsights 0.11.10 pypi_0 pypi
argcomplete 1.12.3 pypi_0 pypi
argon2-cffi 21.3.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
arrow 1.3.0 pypi_0 pypi
asgiref 3.5.2 pypi_0 pypi
asn1crypto 1.5.1 pypi_0 pypi
astor 0.8.1 pypi_0 pypi
asttokens 2.1.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attrs 23.2.0 pypi_0 pypi
awscli 1.27.2 pypi_0 pypi
azure-cli-core 2.29.1 pypi_0 pypi
azure-cli-telemetry 1.0.6 pypi_0 pypi
azure-common 1.1.28 pypi_0 pypi
azure-core 1.25.0 pypi_0 pypi
azure-identity 1.7.0 pypi_0 pypi
azure-mgmt-compute 23.1.0 pypi_0 pypi
azure-mgmt-core 1.2.2 pypi_0 pypi
azure-mgmt-network 19.0.0 pypi_0 pypi
azure-mgmt-resource 20.0.0 pypi_0 pypi
babel 2.11.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
backoff 1.10.0 pypi_0 pypi
bcrypt 3.2.2 pypi_0 pypi
beautifulsoup4 4.12.2 pypi_0 pypi
black 23.3.0 pypi_0 pypi
bleach 5.0.1 pypi_0 pypi
blessed 1.19.1 pypi_0 pypi
bokeh 3.0.1 pypi_0 pypi
boto3 1.34.17 pypi_0 pypi
botocore 1.34.17 pypi_0 pypi
brotlipy 0.7.0 py39h27cfd23_1003
ca-certificates 2022.07.19 h06a4308_0
cachetools 5.3.2 pypi_0 pypi
certifi 2023.11.17 pypi_0 pypi
cffi 1.15.0 py39hd667e15_1
chardet 4.0.0 pypi_0 pypi
charset-normalizer 2.1.1 pypi_0 pypi
click 8.1.7 pypi_0 pypi
cloudpickle 2.1.0 pypi_0 pypi
colorama 0.4.4 pyhd3eb1b0_0
colorful 0.5.4 pypi_0 pypi
comm 0.1.4 pypi_0 pypi
compute-sdk 1.0.3 pypi_0 pypi
conda 4.13.0 py39h06a4308_0
conda-content-trust 0.1.1 pyhd3eb1b0_0
conda-package-handling 1.8.1 py39h7f8727e_0
contourpy 1.2.0 pypi_0 pypi
croniter 2.0.1 pypi_0 pypi
cryptography 38.0.4 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
cython 0.29.26 pypi_0 pypi
daal 2021.7.0 pypi_0 pypi
daal4py 2021.7.0 pypi_0 pypi
darwin 0.7.3 pypi_0 pypi
darwin-deployer 1.0.4 pypi_0 pypi
dask 2022.2.0 pypi_0 pypi
dataclasses-json 0.5.14 pypi_0 pypi
datetime 4.9 pypi_0 pypi
debugpy 1.6.3 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
deepmerge 1.1.0 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
deltalake 0.10.5 pypi_0 pypi
dill 0.3.6 pypi_0 pypi
diskcache 5.4.0 pypi_0 pypi
distlib 0.3.5 pypi_0 pypi
distributed 2022.2.0 pypi_0 pypi
dm-tree 0.1.7 pypi_0 pypi
docutils 0.16 pypi_0 pypi
duckdb 0.9.1 pypi_0 pypi
elasticsearch 7.10.1 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
exceptiongroup 1.2.0 pypi_0 pypi
executing 1.2.0 pypi_0 pypi
faiss-cpu 1.7.4 pypi_0 pypi
fastapi 0.79.1 pypi_0 pypi
fastjsonschema 2.19.1 pypi_0 pypi
filelock 3.8.0 pypi_0 pypi
flatbuffers 1.12 pypi_0 pypi
fonttools 4.47.2 pypi_0 pypi
fqdn 1.5.1 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
fsspec 2023.12.2 pypi_0 pypi
future 0.18.2 pypi_0 pypi
gast 0.4.0 pypi_0 pypi
gitdb 4.0.10 pypi_0 pypi
gitpython 3.1.38 pypi_0 pypi
google-api-core 2.8.2 pypi_0 pypi
google-api-python-client 1.7.8 pypi_0 pypi
google-auth 2.11.0 pypi_0 pypi
google-auth-httplib2 0.1.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-oauth 1.0.1 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
googleapis-common-protos 1.56.4 pypi_0 pypi
gpustat 1.0.0rc1 pypi_0 pypi
greenlet 3.0.0 pypi_0 pypi
grpcio 1.43.0 pypi_0 pypi
gw-dsl-parser 0.1.6a1 pypi_0 pypi
gym 0.23.1 pypi_0 pypi
gym-notices 0.0.8 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
h5py 3.7.0 pypi_0 pypi
heapdict 1.0.1 pypi_0 pypi
httpcore 0.16.3 pypi_0 pypi
httplib2 0.20.4 pypi_0 pypi
httpx 0.23.1 pypi_0 pypi
humanfriendly 9.2 pypi_0 pypi
icebridge 0.0.1.dev0 pypi_0 pypi
idna 3.6 pypi_0 pypi
imageio 2.21.1 pypi_0 pypi
importlib-metadata 7.0.1 pypi_0 pypi
importlib-resources 6.1.1 pypi_0 pypi
ipykernel 6.17.0 pypi_0 pypi
ipython 8.6.0 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
ipywidgets 8.1.1 pypi_0 pypi
iso8601 1.0.2 pypi_0 pypi
isodate 0.6.1 pypi_0 pypi
isoduration 20.11.0 pypi_0 pypi
jedi 0.18.1 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
jmespath 1.0.1 pypi_0 pypi
joblib 1.2.0 pypi_0 pypi
json5 0.9.10 pypi_0 pypi
jsonpatch 1.33 pypi_0 pypi
jsonpath-ng 1.6.0 pypi_0 pypi
jsonpointer 2.4 pypi_0 pypi
jsonschema 4.20.0 pypi_0 pypi
jsonschema-specifications 2023.12.1 pypi_0 pypi
jupyter-ai 1.9.0 pypi_0 pypi
jupyter-ai-magics 2.8.1 pypi_0 pypi
jupyter-client 8.6.0 pypi_0 pypi
jupyter-core 5.7.1 pypi_0 pypi
jupyter-events 0.9.0 pypi_0 pypi
jupyter-server 2.12.2 pypi_0 pypi
jupyter-server-fileid 0.9.0 pypi_0 pypi
jupyter-server-mathjax 0.2.6 pypi_0 pypi
jupyter-server-terminals 0.5.1 pypi_0 pypi
jupyter-server-ydoc 0.8.0 pypi_0 pypi
jupyter-ydoc 0.2.5 pypi_0 pypi
jupyterlab 3.6.6 pypi_0 pypi
jupyterlab-git 0.50.0 pypi_0 pypi
jupyterlab-pygments 0.2.2 pypi_0 pypi
jupyterlab-server 2.25.0 pypi_0 pypi
jupyterlab-widgets 3.0.9 pypi_0 pypi
kafka-python 2.0.2 pypi_0 pypi
keras 2.9.0 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
knack 0.8.2 pypi_0 pypi
kopf 1.35.6 pypi_0 pypi
kubernetes 24.2.0 pypi_0 pypi
langchain 0.0.350 pypi_0 pypi
langchain-community 0.0.7 pypi_0 pypi
langchain-core 0.1.3 pypi_0 pypi
langsmith 0.0.77 pypi_0 pypi
ld_impl_linux-64 2.35.1 h7274673_9
libclang 14.0.6 pypi_0 pypi
libffi 3.3 he6710b0_2
libgcc 7.2.0 h69d50b8_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libstdcxx-ng 9.3.0 hd4cf53a_17
lightgbm 3.3.3 pypi_0 pypi
lightgbm-ray 0.1.8 pypi_0 pypi
locket 1.0.0 pypi_0 pypi
lxml 5.1.0 pypi_0 pypi
lz4 4.0.2 pypi_0 pypi
markdown 3.4.1 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
marshmallow 3.20.2 pypi_0 pypi
matplotlib 3.8.2 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mistune 2.0.4 pypi_0 pypi
mlp-commons 1.0.1 pypi_0 pypi
mlp-fct 2.1.0 pypi_0 pypi
mlp-monitoring 1.0.0 pypi_0 pypi
mmhash3 3.0.1 pypi_0 pypi
modin 0.16.2 pypi_0 pypi
msal 1.18.0 pypi_0 pypi
msal-extensions 0.3.1 pypi_0 pypi
msgpack 1.0.4 pypi_0 pypi
msrest 0.7.1 pypi_0 pypi
msrestazure 0.6.4 pypi_0 pypi
multidict 6.0.4 pypi_0 pypi
mypy-extensions 1.0.0 pypi_0 pypi
nbclassic 0.4.8 pypi_0 pypi
nbclient 0.9.0 pypi_0 pypi
nbconvert 7.2.3 pypi_0 pypi
nbdime 4.0.1 pypi_0 pypi
nbformat 5.9.2 pypi_0 pypi
ncurses 6.3 h7f8727e_2
nest-asyncio 1.5.6 pypi_0 pypi
netifaces 0.11.0 pypi_0 pypi
networkx 3.2.1 pypi_0 pypi
notebook 6.5.2 pypi_0 pypi
notebook-shim 0.2.2 pypi_0 pypi
numpy 1.23.1 pypi_0 pypi
nvidia-ml-py 11.495.46 pypi_0 pypi
oauthlib 3.2.0 pypi_0 pypi
openai 0.28.1 pypi_0 pypi
opencensus 0.11.0 pypi_0 pypi
opencensus-context 0.1.3 pypi_0 pypi
openssl 1.1.1q h7f8727e_0
opentelemetry-api 1.1.0 pypi_0 pypi
opentelemetry-exporter-otlp 1.1.0 pypi_0 pypi
opentelemetry-exporter-otlp-proto-grpc 1.1.0 pypi_0 pypi
opentelemetry-proto 1.1.0 pypi_0 pypi
opentelemetry-sdk 1.1.0 pypi_0 pypi
opentelemetry-semantic-conventions 0.20b0 pypi_0 pypi
opt-einsum 3.3.0 pypi_0 pypi
overrides 7.4.0 pypi_0 pypi
packaging 23.2 pypi_0 pypi
pandas 1.5.1 pypi_0 pypi
pandocfilters 1.5.0 pypi_0 pypi
papermill 2.4.0 pypi_0 pypi
paramiko 2.11.0 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
partd 1.3.0 pypi_0 pypi
pathspec 0.12.1 pypi_0 pypi
petastorm 0.12.0 pypi_0 pypi
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 10.2.0 pypi_0 pypi
pip 22.3.1 pypi_0 pypi
pkginfo 1.8.3 pypi_0 pypi
platformdirs 4.1.0 pypi_0 pypi
ply 3.11 pypi_0 pypi
portalocker 1.7.1 pypi_0 pypi
prometheus-client 0.13.1 pypi_0 pypi
prompt-toolkit 3.0.32 pypi_0 pypi
protobuf 3.19.6 pypi_0 pypi
psutil 5.9.1 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
py-spy 0.3.12 pypi_0 pypi
py4j 0.10.9.5 pypi_0 pypi
pyarrow 14.0.2 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pycosat 0.6.3 py39h27cfd23_0
pycparser 2.21 pyhd3eb1b0_0
pydantic 1.10.13 pypi_0 pypi
pydeprecate 0.3.2 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pygwalker 0.3.7 pypi_0 pypi
pyiceberg 0.4.0 pypi_0 pypi
pyjwt 2.4.0 pypi_0 pypi
pynacl 1.5.0 pypi_0 pypi
pyopenssl 22.0.0 pyhd3eb1b0_0
pyparsing 3.0.9 pypi_0 pypi
pyrsistent 0.18.1 pypi_0 pypi
pysocks 1.7.1 py39h06a4308_0
pyspark 3.2.0 pypi_0 pypi
python 3.9.5 h12debd9_4
python-dateutil 2.8.2 pypi_0 pypi
python-json-logger 2.0.4 pypi_0 pypi
pytorch-lightning 1.6.5 pypi_0 pypi
pytz 2023.3.post1 pypi_0 pypi
pywavelets 1.3.0 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
pyzmq 25.1.2 pypi_0 pypi
ray 2.0.0 pypi_0 pypi
ray-cpp 2.0.0 pypi_0 pypi
ray-lightning 0.3.0 pypi_0 pypi
ray-utils 1.0.2 pypi_0 pypi
raydp 0.5.0 pypi_0 pypi
readline 8.1.2 h7f8727e_1
redis 3.5.3 pypi_0 pypi
redshift-connector 2.0.918 pypi_0 pypi
referencing 0.32.1 pypi_0 pypi
regex 2023.10.3 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rfc3339-validator 0.1.4 pypi_0 pypi
rfc3986 1.5.0 pypi_0 pypi
rfc3986-validator 0.1.1 pypi_0 pypi
rich 13.7.0 pypi_0 pypi
rpds-py 0.16.2 pypi_0 pypi
rsa 4.7.2 pypi_0 pypi
ruamel_yaml 0.15.100 py39h27cfd23_0
s3fs 0.4.2 pypi_0 pypi
s3transfer 0.10.0 pypi_0 pypi
schedule 1.1.0 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scikit-learn 1.1.3 pypi_0 pypi
scikit-learn-intelex 2021.7.0 pypi_0 pypi
scipy 1.9.0 pypi_0 pypi
scramp 1.4.4 pypi_0 pypi
send2trash 1.8.2 pypi_0 pypi
setuptools 67.8.0 pypi_0 pypi
shortuuid 1.0.11 pypi_0 pypi
six 1.13.0 pypi_0 pypi
smart-open 6.1.0 pypi_0 pypi
smmap 5.0.1 pypi_0 pypi
sniffio 1.3.0 pypi_0 pypi
sortedcontainers 2.4.0 pypi_0 pypi
soupsieve 2.5 pypi_0 pypi
sparksql-magic 0.0.3 pypi_0 pypi
sqlalchemy 2.0.22 pypi_0 pypi
sqlglot 18.15.0 pypi_0 pypi
sqlite 3.38.2 hc218d9a_0
stack-data 0.6.0 pypi_0 pypi
starlette 0.19.1 pypi_0 pypi
store 1.0.1 pypi_0 pypi
strictyaml 1.7.3 pypi_0 pypi
tabulate 0.8.10 pypi_0 pypi
tbb 2021.7.1 pypi_0 pypi
tblib 1.7.0 pypi_0 pypi
tenacity 8.2.3 pypi_0 pypi
tensorboard 2.9.1 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorboardx 2.5.1 pypi_0 pypi
tensorflow 2.9.1 pypi_0 pypi
tensorflow-estimator 2.9.0 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.27.0 pypi_0 pypi
termcolor 2.1.0 pypi_0 pypi
terminado 0.17.0 pypi_0 pypi
textwrap3 0.9.2 pypi_0 pypi
threadpoolctl 3.1.0 pypi_0 pypi
tifffile 2022.8.12 pypi_0 pypi
tiktoken 0.5.1 pypi_0 pypi
tinycss2 1.2.1 pypi_0 pypi
tk 8.6.11 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi
toolz 0.12.0 pypi_0 pypi
torch 1.12.0 pypi_0 pypi
torchmetrics 0.10.2 pypi_0 pypi
torchvision 0.13.0 pypi_0 pypi
tornado 6.4 pypi_0 pypi
tqdm 4.66.1 pypi_0 pypi
traitlets 5.14.1 pypi_0 pypi
typeguard 2.13.3 pypi_0 pypi
typer 0.7.0 pypi_0 pypi
types-python-dateutil 2.8.19.14 pypi_0 pypi
typing 3.7.4.3 pypi_0 pypi
typing-extensions 4.9.0 pypi_0 pypi
typing-inspect 0.9.0 pypi_0 pypi
tzdata 2022a hda174b7_0
uri-template 1.3.0 pypi_0 pypi
uritemplate 3.0.1 pypi_0 pypi
urllib3 1.26.18 pypi_0 pypi
uvicorn 0.16.0 pypi_0 pypi
virtualenv 20.16.3 pypi_0 pypi
wasmtime 12.0.0 pypi_0 pypi
wcwidth 0.2.5 pypi_0 pypi
webcolors 1.13 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.3.3 pypi_0 pypi
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
widgetsnbextension 4.0.9 pypi_0 pypi
workflow-sdk 1.0.0 pypi_0 pypi
wrapt 1.14.1 pypi_0 pypi
xgboost 1.7.1 pypi_0 pypi
xgboost-ray 0.1.12 pypi_0 pypi
xyzservices 2022.9.0 pypi_0 pypi
xz 5.2.5 h7b6447c_0
y-py 0.6.2 pypi_0 pypi
yaml 0.2.5 h7b6447c_0
yarl 1.9.4 pypi_0 pypi
ypy-websocket 0.8.4 pypi_0 pypi
zict 2.2.0 pypi_0 pypi
zipp 3.17.0 pypi_0 pypi
zlib 1.2.12 h7f8727e_1
zope-interface 6.1 pypi_0 pypi

### Relevant logs

_No response_