microsoft / microsoft/SynapseML
Lightgbm bug on big dataset
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Describe the bug**
Everything is fine when I train my lightgbm model on a demo dataset(10k samples).
But when I change the demo dataset to a big one(2.7m samples), I got a `Java Runtime Environment Error`
**To Reproduce**
This is my running command:
```
spark-submit --jars $JARS --packages com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-49-659b7743-SNAPSHOT --executor-memory 500G --driver-memory 100G train_on_spark.py
```
This is my code snippets:
```python
config_path = 'config.yaml'
config = Config(config_path)
# train_path is the folder path of tfrecords data
df_train = spark.read.format("tfrecords").option("recordType", "Example").load(config.train_path)
label_col = config.label_column_name
feature_combination = config.feature_combination
to_vector = udf(lambda a: Vectors.dense(a), VectorUDT())
raw_data = df_train.select([to_vector(col).alias(col) for col in feature_combination] + [label_col])
featurizer = VectorAssembler(inputCols=feature_combination, outputCol="features")
columns_to_model = ["features", label_col]
train_data = featurizer.transform(raw_data)[columns_to_model]
params = translate_param_name_for_spark_model(param_dict=config.params, model_conf=config)
if config.task_type == "regression":
model = LightGBMRegressor(**params)
elif config.task_type == "classification":
model = LightGBMClassifier(**params)
checkpoint = model.fit(train_data)
```
**Info**
- MMLSpark Version: mmlspark_2.12:1.0.0-rc3-49-659b7743-SNAPSHOT
- Spark Version: 3.1.1
**Stacktrace**
```
...
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Local rank: 4, total number of machines: 6
[LightGBM] [Info] Local rank: 3, total number of machines: 6
[LightGBM] [Info] Local rank: 5, total number of machines: 6
[LightGBM] [Info] Local rank: 2, total number of machines: 6
[LightGBM] [Info] Local rank: 1, total number of machines: 6
[LightGBM] [Info] Local rank: 0, total number of machines: 6
21/06/01 18:00:11 INFO BlockManagerInfo: Removed broadcast_1_piece0 on node09-cpu:40661 in memory (size: 2.6 KiB, free: 159.8 GiB)
21/06/01 18:00:11 INFO BlockManagerInfo: Removed broadcast_3_piece0 on node09-cpu:40661 in memory (size: 5.1 KiB, free: 159.8 GiB)
21/06/01 18:13:08 INFO PythonUDFRunner: Times: total = 790901, boot = 20, init = 527, finish = 790354
21/06/01 18:13:08 INFO LightGBMRegressor: LightGBM task generating dense dataset with 394097 rows and 4719 columns
21/06/01 18:13:58 INFO PythonUDFRunner: Times: total = 841354, boot = 13, init = 582, finish = 840759
21/06/01 18:13:59 INFO LightGBMRegressor: LightGBM task generating dense dataset with 417444 rows and 4719 columns
21/06/01 18:14:35 INFO PythonUDFRunner: Times: total = 878028, boot = 36, init = 543, finish = 877449
21/06/01 18:14:35 INFO LightGBMRegressor: LightGBM task generating dense dataset with 435436 rows and 4719 columns
21/06/01 18:14:56 INFO PythonUDFRunner: Times: total = 898972, boot = 31, init = 527, finish = 898414
21/06/01 18:14:56 INFO LightGBMRegressor: LightGBM task generating dense dataset with 449106 rows and 4719 columns
21/06/01 18:16:02 INFO PythonUDFRunner: Times: total = 965008, boot = 8, init = 586, finish = 964414
21/06/01 18:16:02 INFO LightGBMRegressor: LightGBM task generating dense dataset with 496211 rows and 4719 columns
21/06/01 18:16:27 INFO PythonUDFRunner: Times: total = 990242, boot = 26, init = 502, finish = 989714
21/06/01 18:16:27 INFO LightGBMRegressor: LightGBM task generating dense dataset with 517806 rows and 4719 columns
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f8f8df3a3a0, pid=534525, tid=0x00007f90908c8700
#
# JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64 )
# Problematic frame:
# C [lib_lightgbm_swig.so+0x103a0] Java_com_microsoft_ml_lightgbm_lightgbmlibJNI_doubleArray_1setitem+0x0
#
# Core dump written. Default location: /gxr/liuyong/Projects/reorg_CPI/CPI-prediction/core or core.534525
#
# An error report file with more information is saved as:
# /gxr/liuyong/Projects/reorg_CPI/CPI-prediction/hs_err_pid534525.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted (core dumped)
```
[AB#1205746](https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_workitems/edit/1205746)
Contributor guide
Research direction
Start with train_on_spark.py and config.yaml, then reproduce the failure using the provided spark-submit command and large TFRecords dataset. Inspect the native crash at lib_lightgbm_swig.so and the generated hs_err_pid534525.log; done means identifying and resolving the SIGSEGV during large-dataset training, with the job completing successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100