[BUG]Inconsistent Training AUC after Saving/Loading Parquet Data

Nobody has claimed this yet.

### SynapseML version

synapseml_2.12:1.0.11

### System information

- **Language version** (e.g. python 3.8, scala 2.12): Scala 2.12
- **Spark Version** (e.g. 3.2.3): Spark 3.5.0
- **Spark Platform** (e.g. Synapse, Databricks): Databricks

### Describe the problem

I'm writing to report a critical reproducibility issue we've encountered with LightGBM training in Synapse ML that significantly impacts model reliability in production environments. During our model development process for a binary classification task with highly imbalanced data (initial positive:negative sample ratio 1:3804 downsampled to 1:1000), we've observed inconsistent training results that directly correlate with whether we use freshly processed data versus reloaded Parquet files.

The core issue manifests as dramatically different training AUC when using logically identical datasets:
**1. When training directly on the downsampled DataFrame before any persistence, we obtain expected training AUC values around 0.92 with validation AUC at 0.91
2. When saving this identical dataset as Parquet then reloading it for training, the training AUC inexplicably jumps to 0.99 (near-perfect scores indicating severe overfitting), while validation AUC remains stable at 0.91**

This inconsistency persists despite:
1. Using the same cluster configuration and notebook environment
2. Verifying identical data content through schema validation and row-count checks
3. Set seed=777 and deterministic=True during the training process
4. Multiple attempts to force partition uniformity through repartition()

Could you help us try to figure it out? The inconsistency indicates there might be some error happens inner our API.

### Code to reproduce issue

```

# define downsampling function
def random_undersampling(train_data, val_data, negative_positive_ratio=10, seed=21):
"""
random sample methods:

Args:

negative_positive_ratio: ratio of positive samples, normally 3-5
n_samples: sampling times
"""
# get positive samples
pos_data = train_data.filter(F.col("conversion_flag") == 1)
neg_data = train_data.filter(F.col("conversion_flag") == 0)

# calculate sampling ratio
pos_count = pos_data.count()
target_neg_count = pos_count * negative_positive_ratio
sampling_fraction = target_neg_count / neg_data.count()

# random sampling
sampled_neg = neg_data.sample(False, sampling_fraction, seed)
# merge
sampled_train = pos_data.unionAll(sampled_neg)
sampled_train_shuffle = sampled_train.orderBy(rand())

return sampled_train_shuffle, val_data

# run the down sampling function
sampled_train, val_data = random_undersampling(
train_data=train_data,
val_data=val_data,
negative_positive_ratio=1000,
seed=299
)

sampled_train.write.format("parquet") \
.mode("overwrite") \
.option("compression", "snappy") \
.save("xxxxx")

sampled_train_reloaded = spark.read.parquet("xxxxx")

from pyspark.sql.functions import lit
from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler
from synapse.ml.lightgbm import LightGBMClassifier
import numpy as np
from pyspark.ml.evaluation import BinaryClassificationEvaluator, RegressionEvaluator
# check the partition distribution difference pre/after reload the data

# Create VectorAssembler
lgbm_assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
lgbm = LightGBMClassifier(
featuresCol="features",
labelCol="conversion_flag",
objective='binary',
isProvideTrainingMetric=True,
numIterations=685.0,
learningRate=0.09801426202661519,
numLeaves=91.0,
lambdaL1=0.7071587286417129,
lambdaL2=0.13903538813177022,
maxDepth=9.0,
minSumHessianInLeaf=0.48631862969629164,
featuresShapCol="shap_values",
featureFraction=0.5229432743203031,
baggingFraction=0.9417315628234632,
baggingFreq=9.0,
passThroughArgs="scale_pos_weight=4.0",
deterministic=True,
seed=777,
baggingSeed=521,
useBarrierExecutionMode=True,
dataTransferMode="streaming"
)

evaluator_auc = BinaryClassificationEvaluator(labelCol="conversion_flag", rawPredictionCol="rawPrediction", metricName="areaUnderROC", numBins=0)
evaluator_pr = BinaryClassificationEvaluator(labelCol="conversion_flag", rawPredictionCol="rawPrediction", metricName="areaUnderPR", numBins=0)
evaluator_rmse = RegressionEvaluator(labelCol="conversion_flag", predictionCol="prediction", metricName="rmse")

# lgbm.setPassThroughArgs("print_every_n_iterations=5")
pipeline = Pipeline(stages=[lgbm_assembler, lgbm])
model = pipeline.fit(sampled_train)
predictions_train = model.transform(sampled_train)
predictions = model.transform(val_data)

train_auc = evaluator_auc.evaluate(predictions_train)
train_auprc = evaluator_pr.evaluate(predictions_train)
val_auc = evaluator_auc.evaluate(predictions)
val_auprc = evaluator_pr.evaluate(predictions)

# Make predictions on val set
print('-----------------------')
print('training AUC:', train_auc)
print('training AUPRC:', train_auprc)
print('validation AUC:', val_auc)
print('validation AUPRC:', val_auprc)

#%%
# then re-run the model use the sampled_train_reloaded
# Create VectorAssembler
lgbm_assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
lgbm = LightGBMClassifier(
featuresCol="features",
labelCol="conversion_flag",
objective='binary',
isProvideTrainingMetric=True,
numIterations=685.0,
learningRate=0.09801426202661519,
numLeaves=91.0,
lambdaL1=0.7071587286417129,
lambdaL2=0.13903538813177022,
maxDepth=9.0,
minSumHessianInLeaf=0.48631862969629164,
featuresShapCol="shap_values",
featureFraction=0.5229432743203031,
baggingFraction=0.9417315628234632,
baggingFreq=9.0,
passThroughArgs="scale_pos_weight=4.0",
deterministic=True,
seed=777,
baggingSeed=521,
useBarrierExecutionMode=True,
dataTransferMode="streaming"
)

evaluator_auc = BinaryClassificationEvaluator(labelCol="conversion_flag", rawPredictionCol="rawPrediction", metricName="areaUnderROC", numBins=0)
evaluator_pr = BinaryClassificationEvaluator(labelCol="conversion_flag", rawPredictionCol="rawPrediction", metricName="areaUnderPR", numBins=0)
evaluator_rmse = RegressionEvaluator(labelCol="conversion_flag", predictionCol="prediction", metricName="rmse")

# lgbm.setPassThroughArgs("print_every_n_iterations=5")
pipeline = Pipeline(stages=[lgbm_assembler, lgbm])
model = pipeline.fit(sampled_train_reloaded)
predictions_train = model.transform(sampled_train_reloaded)
predictions = model.transform(val_data)

train_auc = evaluator_auc.evaluate(predictions_train)
train_auprc = evaluator_pr.evaluate(predictions_train)
val_auc = evaluator_auc.evaluate(predictions)
val_auprc = evaluator_pr.evaluate(predictions)

# Make predictions on val set
print('-----------------------')
print('training AUC:', train_auc)
print('training AUPRC:', train_auprc)
print('validation AUC:', val_auc)
print('validation AUPRC:', val_auprc)

```

### Other info / logs

```
this one is before save/reloading:
training AUC: 0.9223300884299345
training AUPRC: 0.8062838605214455
validation AUC: 0.916019539596558
validation AUPRC: 0.7558880508858233

this one is after save/reloading:
training AUC: 0.9999999963765849
training AUPRC: 0.9999963845879573
validation AUC: 0.9082770248549157
validation AUPRC: 0.7598785340365525
```

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [ ] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [ ] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

The issue names no repository files or tests; start with the LightGBMClassifier and Pipeline.fit entry points in the supplied reproduction, then compare training on the original DataFrame with training after Parquet save and reload. Trace how partitioning, row order, and data transfer are handled, and consider the work complete when the cause is isolated and equivalent inputs produce reproducible training behavior.

Written by the indexing model from the issue text.