microsoft / microsoft/SynapseML

Using TrainClassifier with class weights (including ClassBalancer)

Open
#555 3 comments 0 reactions 1 assignee View on GitHub

@imatiach-msft is already working on this.

Since Apr 25, 2019.

area/lightgbm bug
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

Hi,

I am having a problem with using the TrainClassifier when the label's classes have weights (including instance weights generated by the MMLSpark Class Balancer class).

I am using Pyspark 3 on Spark 2.2.0.

The non-MMLSpark way of creating a classifier model with weights, for a table with labels defined by a variable label_col, feature column defined by a variable col, and weight column weight, would be something like:

`LogisticRegression().setWeightCol("weight").setLabelCol(label_col).setFeaturesCol(col)
`
I tried different combinations to do it using MMLSpark:

**#1:**
```
base_model = LogisticRegression().setWeightCol('weight').setFeaturesCol(col).setLabelCol(label_col)
TrainClassifier(model=base_model)
```

will lead to the following error when training the model:

```
Py4JJavaError: An error occurred while calling o17763.fit.
: java.util.NoSuchElementException: Failed to find a default value for labelCol
```

presumably because the TrainClassifier constructor requires setLabelCol to be set explicitly.

**#2:**

```
base_model = LogisticRegression().setWeightCol('weight').setFeaturesCol(col)
TrainClassifier(model=base_model).setLabelCol(label_col)
```

will lead to a model with 2 coefficients for a single feature and 1 intercept. Presumably, because the weight column was used as a feature, and the feature column set by the base method's "setFeaturesCol" was ignored by TrainClassifier.

**#3:**

```
base_model = LogisticRegression().setWeightCol('weight')
TrainClassifier(model=base_model).setLabelCol(label_col).setFeaturesCol(col)
```

will lead to the following error when training the model:

```
Py4JJavaError: An error occurred while calling o17925.fit.
: org.apache.spark.sql.AnalysisException: Reference '__' is ambiguous, could be: __#13901, __#32151.;
```

presumably because the TrainClassifier constructor changes the names of the features, and the setFeaturesCol doesn't work properly it seems.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.